Hacker News Reader

Comment by sargun | Hacker News Reader

The problem I have with sqlite for being a general purpose file format is that it’s difficult to take a sqlite file in any language and work with it without linking against code.

I kind of wish that there were native implementations of sqlite in every language.

View on HN

WASM is really interesting here.

https://github.com/sql-js/sql.js is SQLite complied to WASM, which means you can run it not just in the browser but also in any WASM environment.

https://github.com/wasmerio/wasmer-python lets you execute WASM compiled binaries within Python.

https://github.com/wasmerio/wasmer-ruby handles Ruby.

So it should be possible to take the WASM compiled SQLite and run it in other languages.

This is a thing that really excites me about WASM: as a mainly-Python programmer it has the potential to give me access to a much wider range of tools.

Nice links. (Hi again)

Bring full SQL select, join, and order in datasette into the browser, perhaps?

What's the problem of linking against SQLite? In my book it's much better because:

- SQLite is coded in C, so performance and weight are going to be equal or better than any other reimplementation.

- Having one code base means no compatibility issue between implementations

- Also SQLite's codebase is so heavily tested and its track record is so good that I really don't see the point.

I'll give a couple examples of where I've needed a K/V database, but linking caused problems:

* We have a sandboxed Javascript runtime inside of the JVM that can only do restricted work. IT would be nice to be able to use something like a SQLite file to deliver configuration to this Javascript runtime

* There are environments where you cannot install any native code -- like serverless providers that require your software to be pure Java / Javascript / Python

* Cross-language debugging can be very difficult when something goes wrong

* Erlang's FFI model is either "blocking and fast" (it runs the FFI call inline, and halts preemptive multitasking), or "slow and safe" (dirty NIFs, where it runs FFI calls on a dedicated thread.) This can get really ugly, really quickly.

https://www.zdnet.com/article/google-chrome-impacted-by-new-...

He didn't say it never has bugs. It does have a great track record and Dr. Hipp fixes things quickly when found.

If there were implementations of SQLite in other languages, do you believe chrome would have used a different implementation?

I don't know! But if this sort of thing can happen in Chrome, I think it's a pretty strong counterargument to "let's standardize formats in terms of a C implementation of a thing that parses a complicated language, among many other things".

XML has been a source of countless vulnerabilities, yet we continue to use it as a standard.

XML has its own problems but 'has a single canonical implementation in a big hairball of C' is not among them.

Non-C implementations of XML don't have countless vulnerabilities. There's XXE and schema requests which are language independent, but otherwise... it's ok.

FWIW I've used golang linking against sqlite in a couple of my projects. One issue is that linking through cgo is really slow, due to the fact that the runtime has no idea anything about types. Probably OK if you're just doing basic queries at a reasonable rate; but if you implement a custom callback, suddenly all the marshaling back and forth takes a huge amount of time.

Not a deal-breaker; but a native implementation of sqlite in golang would be a lot nicer.

> Not a deal-breaker; but a native implementation of sqlite in golang would be a lot nicer.

CGo-free SQLite for Linux: https://godoc.org/modernc.org/sqlite

In this particular case though you have to weigh linking against SQLight versus zlib and libxml.

Native implementations of both those are available for many languages, though. ~no-one is using libxml with Java, say.

Yes.

Possibly if this was done however, we'd get native sqllite file format read/write libraries in most major languages.

Might overall be good. Sqllite-as-file-format isn't necessarily a bad idea if this infrastructure exists.

Why can't you just have your language call to SQLite via the command line?

One reason SQLite is so useful for this kind of application is that once you've linked it in any SQL call can be treated as a C function call.

Many Small Queries Are Efficient In SQLite: https://www.sqlite.org/np1queryprob.html

Round-tripping queries via shelling out to the SQLite binary would lose much of that advantage.

So efficiency?

Reply on news.ycombinator.com