Migration of the build system to autosetup
sqlite.org
sqlite.org
Richard Hipp, of sqlite fame, has announced that the sqlite project plans to reimplement its canonical build process in the 3.48 release cycle. The project will, if all goes as planned, be moving away from the GNU Autotools and towards Steve Bennett's Autosetup, which we've used to good effect since 2011 in SQLite's sibling project, the Fossil SCM.
The primary motivation behind this HN post is to draw eyes to what Richard points out in the linked forum post: this port _will_ break some folks' build processes and we'd like to know, in advance, what we'll be breaking so that we can make that transition as painless as we can. If you do "unusual" things with the SQLite build, please let us know about it (preferably in the linked-to forum, as that's less likely to be overlooked than comments on HN are, but we'll be following this thread for at least a few days for those of you who would prefer to voice your thoughts here).
recipe/build.sh: https://github.com/conda-forge/sqlite-feedstock/blob/main/re...
recipe/build.bat: https://github.com/conda-forge/sqlite-feedstock/blob/main/re...
Is there a [multi-stage] Dockerfile to build, install and test SQLite with the new build system?
Is there SLSA signing for the SQLite build artifacts? There should be a way to sign OBS Open Build Service packages with sigstore.dev's Cosign to Rekor, too IIUC.
slsa-framework/slsa-github-generator: https://github.com/slsa-framework/slsa-github-generator
Aside from the cross-compilation, which is most definitely part of the process but has not yet been explored in depth[^1], nothing in your build immediately sticks out at as problematic, but i've bookmarked it for future reference during the port.
The .bat build (Windows) will not change via this port. Only Unix-like environments will potentially be affected.
> Is there a [multi-stage] Dockerfile to build, install and test SQLite with the new build system?
Not that the sqlite project maintains, no. None of us use docker in any capacity. We work only from the canonical source tree and we like to think that it's easy enough to do that other folks can too.
> Is there SLSA signing for the SQLite build artifacts?
That's out of scope for, and unrelated to, this particular sub-project.
> slsa-framework/slsa-github-generator: https://github.com/slsa-framework/slsa-github-generator
says:
> This repository contains free tools to generate and verify SLSA Build Level 3 provenance for native GitHub projects using GitHub Actions.
We don't use github except to post a read-only mirror of the canonical source tree.
[^1]: Autosetup has strong support for cross-compilation, it just hasn't yet been tested in the being-ported build process.
It's probably possible to trigger gh actions or gl pipelines builds (container, command) with git repo mirroring.
datasette-lite depends on a WASM build of SQLite; what about WASM and e.g. emscripten-forge and micromamba?
Edit
emscripten-forge/recipes/recipes_emscripten/SQLite/build.sh: https://github.com/emscripten-forge/recipes/blob/main/recipe... ... recipe.yml says 2022: https://github.com/emscripten-forge/recipes/blob/main/recipe...
That's all out of scope for, and unrelated to, this build port.
> It's probably possible to trigger gh actions or gl pipelines builds (container, command) with git repo mirroring.
We maintain our own SCM, which keeps us far removed from the github ways of doing things. None of the SQLite project members actively use github beyond posting the read-only project mirror and occasionally cloning other projects and/or participating in tickets, and that's unlikely to change in any foreseeable future.
> emscripten-forge/recipes/recipes_emscripten/SQLite/build.sh:
Thank you - bookmarked for later perusal and testing.
What about extensions; does the new build port affect extensions?
/? hnlog sqlite (190+ mentions)
From https://news.ycombinator.com/item?id=40837610#40842365 :
- > There are many extensions of SQLite; rqlite (Raft in Go,), cr-sqlite (CRDT in C), postlite (Postgres wire protocol for SQLite), electricsql (Postgres), sqledge (Postgres), and also WASM: sqlite-wasm, sqlite-wasm-http, dqlite (Raft in Rust),
awesome-sqlite
It won't, in any way, affect how third parties use the conventional project headers and amalgamated sources. It only has the potential to affect folks who use the "./configure && make" build process, as differences between the Autotools and Autosetup make it impossible to retain 100% semantic compatibility with the build processes (for reasons described one link removed from the forum post this HN post links to).
It turns out that that's not _entirely_ true now: Richard got jimsh working on Windows, which means that nmake-driven Windows builds can now build SQLite (sans the testing suite) without requiring an extra installation of TCL.
Duplicate threads are a plague ;).
Posting a second time will only lead to chaos as i try to juggle multiple threads while continuing work on the posted-about build port.
Personally I'm fine with autoconf (not with automake and libtool!), but it does require some maintenance. I've never made the jump to CMake, but autosetup looks like a viable alternative. It is nice to see a Tcl application with an included interpreter!
A link in the original post leads to a description of the motivation for the port. TL;DR: we've used Autosetup since 2011 in SQLite's sibling, the Fossil SCM, and we quite like it.
> Or are there more fundamental reasons to prefer it over cmake?
CMake is an out-of-tree tool, which is something the project tries to avoid at all costs for portability and maintainability reasons. GNU Autotools lives halfway in-tree and halfway out-of-tree. It's ubiquitous but also often problematic to upgrade, and is subject to breakage when project developers have different versions of the Autotools installed on their systems. Richard very specifically keeps an ancient version of Autotools on his system to reduce that pain, but that's not a satisfying situation to be in.
Autosetup, on the other hand, lives entirely in the project's source tree, meaning we have not only consistency across all developers' machines, but also complete control over when to upgrade (or not) the tools. Over the past 13ish years of using Autosetup in the Fossil SCM we have never had any significant upgrade-related pains.
Is this relevant though? Portability is not dictated by whether you vend a tool within your project tree, and maintainability is only remotely relevant if you're planning on customizing the tool instead of the scripts executed by the tool, which would be silly.
I mean, there is no wrong in just coming forward and stating that you're already using a build system in another project and you're hoping to lower maintenance costs by standardizing on a single build system that you're already using. All other arguments sound dubious at best.
> Is this relevant though?
For this project and its day-to-day maintenance, unequivocally.
One of Richard's philosophies is "freedom means being able to take care of yourself," and that philosophy permeates all of his software projects, which eschew external dependencies because every single one reduces the project's level of freedom. Every time a third-party tool breaks, or introduces new, incompatible behavior as part of an upgrade, dependents of that tool suffer[^1]. All of Richard's projects have a strong culture of avoiding that suffering by avoiding third-party dependencies where at all possible, to the point of sometimes reinventing the wheel to extreme degrees (like implementing his own SCM).
That aspect aside, nobody in the SQLite project uses CMake, so it would never occur to us to migrate the build process to it. We write our own makefiles by hand _and we like it that way_. (Sidebar: Autosetup neither writes our makefiles nor introduces new syntax for doing so, in contrast to CMake.)
> I mean, there is no wrong in just coming forward and stating that you're already using a build system in another project and you're hoping to lower maintenance costs by standardizing on a single build system that you're already using. All other arguments sound dubious at best.
This isn't about standardizing on one build process, but about simplifying the lives of the developers and future maintenance of the project. We can't do that by porting to tools we don't otherwise use. If we weren't using Autosetup in other trees, and weren't happy with it, we wouldn't have even considered undertaking this effort and facing the downstream upheaval it will undoubtedly cause. We know, however, from 13 years of experience with it, that this tool does what we want, is low friction, and causes us an amount of grief which closely approaches zero[^2]. Without our collective background with Autosetup, we wouldn't have even considered porting away from the Autotools because that would be a case of "out of the frying pan and into the oven."
[^1]: As a concrete example: the SQLite project's WASM build is 100% dependent on the Emscripten toolchain for the simple reason that there is no feasible alternative. A handful of times over the past two years we have had to accommodate incompatible changes when upgrading Emscripten. That's hassle nobody wants. Obviously, software has to evolve, but being able to embed our build tools (Autosetup) directly in the source tree gives us control over the pace of that evolution. No system-wide upgrade is going to pull it out from under us, nor will it behave differently on downstream folks' machines.
[^2]: You won't ever hear me say that _any_ software is 100% grief-free, but autosetup is as close to it as, e.g., the bash shell.
Insisting on personal assertions does not pass as a justification, particularly one based on such specious reasoning.
> One of Richard's philosophies is "freedom means being able to take care of yourself," (...)
That's fine. Irrelevant to the discussion, though. If you're really really concerned with somehow losing access to a build tool managed by a third-party, nothing prevents you from storing the tool anywhere, including in a separate Git repository.
Just say it's personal taste and technical merits play no role in the decision. All these wild claims just sound silly.
Reducing long-term maintenance burden and simplifying the developers' work is arguably the best technical merit we could hope for.
> All these wild claims just sound silly.
And, with that, i'm stepping out of this sub-thread.
For example, suppose that the CMake developers, due to last-straw Unicode bugs, lose sanity and decide that from next version (in a few weeks) an initial BOM and UTF-8 encoding in all files will be mandatory: are you ready to check and adjust all your source files and review your text editors (or to fork CMake)?
More realistic scenarios include users discovering novel CMake bugs and changed defaults after updating CMake and developers discovering useful things that are impossible with CMake because of old bugs and limitations.
Is it really desirable to have complex build tools be part of the project, especially after the xz-utils takeover?
That's a fair question, but i'm confident that in this case such a risk is simply not there. We've interacted with the maintainer of Autosetup and JimTCL (the same guy) for more than a decade and know he runs a proverbial "tight ship."
Autosetup has two major components:
- TCL-based scripts to process configure-style scripts and work with file templates in a way nearly identical to autotools (e.g. Makefile.in uses the same syntax).
- JimTCL is a TCL implementation which lives in the tree, allowing folks who won't have the canonical TCL installed to run the build.
Steve Bennett maintains both of those and is extremely conscientious about what goes into his trees.
Over the years, as we've updated our in-tree copy in the Fossil SCM, it's become habit to peruse the diffs of the upgraded files, comparing them to our previous install. In the unlikely case that someone were to sneak some mischief in, i'm fairly confident that we would catch it, at the latest, at that point (before checking in that change to our project).
That's most definitely a big part of it, but see my other response in this sub-thread for other aspects which are equally important for the project.