Hacker News Reader

Hacker News

Comment by acqq | Hacker News Reader

Parent Full thread

acqq

Moreover, if it's about cross-compilation, why not using zig?

https://zig.news/kristoff/building-sqlite-with-cgo-for-every...

View on HN

It's not just about cross compilation.

There's also debuggability and other aspects of development.

It's an anecdote, but I've had a better experience debugging issues while developing SQLite extensions and VFSes for this driver than I've had in other "cross runtime" / "bindings" work I've done before (with Cgo, JNI, P/Invoke).

I would need to read more about that to understand how it could be a problem. You see, I've literally spend decades using C and I can never imagine how adding that what I understand to be huge dependencies like wazero.io and executing huge wasms binaries made from original C sources can be more debuggable than debugging a directly compiled native code -- I'm just disclosing my bias, and I want to be clear about it, I'm certainly not doubting your "better experience."

Edit: have you tried that zig workflow for cross-compiling?

Edit2: and I'm also aware that zig also uses wasm binary and its own wasm2c, written from scratch, for its own bootstrapping:

https://ziglang.org/news/goodbye-cpp/

I don't mean debugging the C bits (although having a stack trace and an exact line number when I cause SQLite to crash is rather nice).

I mean developing a complex wrapper/binding that involves Go calls C, which then calls back to Go, which again calls C, and putting breakpoints on the Go bits multiple layers deep, and being able to inspect the full stack of those Go bits.

Last I've tried with Cgo, stuff has a high probability of hanging your entire process (e.g. if the C library acquired some lock or something).

There's also the permanent fear that (bugs, API misuse…) leads to the C library corrupting your Go memory, which is much harder to do if the C bits run in a sandbox. SQLite is "imune" to bugs, but not API misuse.

Tbh, I haven't had a better experience with JNI, JNA, PInvoke…

And yes, I've looked at zig, even used it initially to build the WASM blob. I actually think it'd be a worthy endeavour to build a better Cgo driver. But having Cgo and WASM versions of this in the same repo is just too much effort at this point (I've considered it).

I'm curious what issues you had with P/Invoke. I use SQLite's C API from C# using P/Invoke rather than using a high-level .NET library. I've implemented virtual table modules, VFSes, application-defined SQL functions, you name it. I even reach into SQLite and call their internal tokenizer directly. The Visual Studio debugger was key to getting this done productively. With C#'s high level of interoperability with C, I can't imagine a similar "SQLite transpiled to C#" project gaining traction in our ecosystem. I think C# stands out from the crowd on interop with C libraries.

Issue one: someone else was employed professionally to develope those bindings for you. :)

Once that's done correctly and at a high quality, your life is easier.

Developing those bindings is another story. But PInvoke is one of the best development stories, I'm not dinging on it.

No, I wrote my own bindings to the C API with P/Invoke. I _don't_ use any other high-level library in .NET. Previously I used C++/CLI, but I migrated it all to vanilla C# with P/Invoke. I have significant experience with the SQLite C API (which is why I don't bother with other people's bindings). With P/Invoke it was a pretty straightforward experience, and specifically the debugger worked great and was crucial to getting it working.

I really think that C# with P/Invoke and the Visual Studio debugger is a cut above the rest.

Oh, great! I assumed you meant the officially supported System.Data.SQLite bindings. Do you have a link or is this internal?

The meat of the P/Invoke code is in here: https://github.com/electroly/sqlnotebook/tree/master/src/Sql...

The parent directory includes code that uses it. I'm most proud of this SQLite virtual table module that proxies queries to remote ADO.NET connections, allowing you to write joins directly between local SQLite tables and remote SQL Server tables. https://github.com/electroly/sqlnotebook/blob/master/src/Sql...

This is exposed to the user as a "LINK" option on the "IMPORT DATABASE" statement: https://sqlnotebook.com/import-database-stmt.html

I've also got a generic virtual table module that lets me easily write table-valued functions in C#: https://github.com/electroly/sqlnotebook/blob/master/src/Sql...

I don't have too many table-valued functions yet but one example is the "LIST_XLS_WORKSHEETS" function: https://sqlnotebook.com/list-xls-worksheets-func.html

I have my own implementation of SQLite's grammar so I can embed it inside my larger grammar which adds things like DECLARE, IF, and WHILE: https://github.com/electroly/sqlnotebook/blob/master/src/Sql...

The goal is to provide various "supercharged" features to base SQLite by taking advantage of all the extension points I can. I wish some went further; in particular the virtual table API doesn't "push down" enough of the original query to allow the module to avoid doing N+1 queries in some cases.

Always happy to see people using SQLite in unique ways!

Oh I wanted to implement a virtual table to do the same thing in Go (access a remote DB, with where clause push down).

Will have a look at your code!

P/Invoke is similar to how many compiled languages used to bind to other compiled languages for many years, e.g. Modula-2, BASIC and Pascal dialects.

Somehow it became common to do it in more complex ways, and the return to AOT compilation is rediscovering it.

The way VS debugging allows for mixed language debugging is only comparable with Netbeans/Eclipse, but neither JNI nor Panama are as good as P/Invoke.

Nothing ever comes close to P/Invoke in ease of use* (while remaining practically zero-overhead if you don't need to marshal strings). JNI is on the opposite side of the spectrum and should never be even compared as it is at least ten times worse developer experiences (and equally higher overhead).

*in garbage-collected languages, otherwise it's pretty much like calling C from Rust.

I'm sorry. The debugging experience of trying to whip up PInvoke wrappers for the COM library for Kinect 2 that worked with Unity Engine's awful version of Mono was charring.

Also had “fun” building some Xamarin Android PInvoke-to-JNI bindings. I just died a little bit remembering this.

No language has good interop story with Android unless there's no interop and you are using Java/Kotlin.

As for the first argument - this really does not represent the average experience in the current day ecosystem. Kinect 2 was a thing 9 years ago and together with Unity it's pretty much an exotic scenario. Unity in general is really not a good example until they get Unity 6 out and move over to vanilla .NET.

To be clearer to those who don't read what's linked but reflexively downvote:

zig is not only the language, it's also a build system that compiles C (also in Go CGO use scenarios) better than the default C compilers, allowing order of magnitude easier cross compilation. It's described in the link I've already provided exactly by compiling SqLite for its use in Go!

Reply on news.ycombinator.com