Richard Hipp Speaks Out on SQLite (2019) [pdf]
sigmodrecord.org
sigmodrecord.org
Q: If you magically had enough extra time to do one additional thing at work that you’re not doing now, what would it be?
A: Oh, I’ve got a long list. But my #1 thing right now I think would be a new version control system which the working name is Fit, it’s a combination of Fossil with Git. Uses the Fossil user interface but it uses the low-level file format of Git. So that then you can work with Fossil’s interface but push and pull to legacy Git users. And I think that would be huge.
The development happens on its mailing list, but there is also an Heptapod (a gitlab fork modified to also support mercurial) instance here: https://foss.heptapod.net/mercurial/tortoisehg
Interesting enough, almost all my work nowadays consists of interacion with git remotes. But my local clone has always been mercurial via hg-git.
Although usually what seems to happen is people only use one linear history and never branch anything. I'm not sure if that's because Git is hard, or just because nobody wants to learn about branches and stuff in any system if they don't have to.
In any case it works perfectly fine and is an amazing step up compared to no VCS.
I don't think it's right to call that "just a new porcelain".
I have no clue how Hipp thinks to implement it, much less in a way that allows push and pull to legacy Git users.
Git is primarily a file format. There's also a canonical set of low-level command-line utilities. But you can create whatever interface you want. Most programmers and GUIs happen to use the canonical command-line tools, but that's hardly necessary. Fit would keep the necessary part of Git (the file format), and provide a different human interface.
I'm surprised that's the #1 magic-wand project for anyone. I'm not trying to yuck his yum, but am just surprised.
But the larger elephant in the room for any such effort is that Fossil's mental model is, by choice, way different from git. If they just want a new ui that isn't as "git sux," surely a new cli to kid-gloves git would be less work than trying to reinvent a whole new version control system, and that goes 1000x for trying to maintain wire compat with every git server in the world
> But the larger elephant in the room for any such effort is that Fossil's mental model is, by choice, way different from git.
How? I've been glancing at docs but I don't see much difference, still a DAG of commits that are named after its hash like almost every other DVCS
From onlooker perspective the killer feature seems to be integration with project management tools vs "just" being a VCS
It seems like Dr. Hipp is frustrated that Fossil hasn’t had mainstream adoption, and that git has, he’s talked about it quite a bit in Fossil docs and here on HN. Given that SQLite is very stable and adoption is an absolutely runaway success, I’m not too surprised his wish list now turns to Fossil.
> Fossil’s mental model is, by choice, way different from git.
Is that accurate? It does come with a ton of extra features, but seems like the core version control model is quite similar conceptually, and differs more on implementation (SQLite backing instead of file system) and workflow dogma (Fossil’s moralizing of rebase and commit graph edits). The Fossil docs even start with “The feature sets of Fossil and Git overlap in many ways.” Fossil was released after but within a year of git being released, the timeline and docs and comments from the authors generally suggest it’s a reaction to git, taking the good parts and trying to, in the authors’ eyes, improve and fix their perceived problems with git.
We'll start with Fossil not tolerating `git rebase` or `git commit --amend`: https://www.fossil-scm.org/home/doc/trunk/www/fossil-v-git.w... and the number of repos I've seen where PRs don't land without someone chirping about "please rebase your branch"
"Fossil was designed for [...] just four programmers, and 64% of it is from the lead developer alone. [They] know each other well and interact daily." https://www.fossil-scm.org/home/doc/trunk/www/fossil-v-git.w... makes one wonder what kind of awesomeness would ensue if running `fit clone https://github.com/chromium/chromium.git`
and its related topic: "Fossil is not designed for drive-by contributions" and "Where Git encourages siloed development, Fossil fights against it.": https://www.fossil-scm.org/home/doc/trunk/www/fossil-v-git.w... -- I would be really sad to have all my working changes auto-pushed by the tool, although the https://github.com/martinvonz/jj#readme is also in that spirit, where almost every operation is a `git add . && git commit -mwip`
Fossil is `git push --mirror` by default: https://www.fossil-scm.org/home/doc/trunk/www/fossil-v-git.w... versus `git push origin HEAD` which is what most people think of with "git push"
Then, as one of the obvious actual technical issues: Fossil is SHA-3: https://www.fossil-scm.org/home/doc/trunk/www/fossil-v-git.w...
BTW, Fossil’s comparisons to git (especially rebase and rewriting) have always been hyperbolic. It’s improving slowly over the years, however the fundamental problem is still there: the Fossil team is presumptuously stating git’s intentions under Fossil’s overtly moral framework. Git devs have never stated any intention “to record what the development of a project should have looked like had there been no mistakes.” That’s the Fossil team framing things to sell the idea behind Fossil. Ironically, the Fossil team is attempting to rewrite the history of git development by using such language. I think the reason behind Fossil’s lack of adoption isn’t because git is entrenched, I suspect it’s because very few people care about having the ability to record in stone every single commit no matter how messy, most people enjoy having the ability to clean up after the fact. (Not to mention that the idea of capturing and preserving an exact “history” is misguided and fundamentally impossible; it’s already up to me what I put in a commit in the first place, and Fossil is providing dis-incentives to commit early and often, undermining the safety reasons to have version control in the first place.)
I think trunk-based, warts and all, is probably the best method for a team that isn't extremely async. Which, I think, makes it useful for a large number of projects.
Is that quote talking about the development of git, the VCS itself, or about what people use git for; are your "Git devs" the people who develop git, or those who use it to develop other software?
As I understand it, some git workflows (that is, workflows using git) are heavy into rewriting history and eliminating branches (or never starting them in the first place) and stuff, but the developers of git never foresaw or mandated that. So my own answer to my question above would be ~"The statement is about both: The developers of git never said that you must use git to straighten out and prettify the history "of a project." Am I reading you correctly in that this is what you too are saying?
Because if you are, then I agree: Sounds like Dr Hipp is exaggerating a little, trying to imply that one particular kind of workflow were (almost) mandatory in git. Creates a bit of a strawman, because that kind of workflow is easier to argue against than arguing against a product that allows that kind of workflow and other kinds.
How's that "feature" of fossil ? You can just decide to not allow it in Git as you see fit, and it is something that is useful once you know how to.
This whole set basically sounds like "Git allows you to do all that but fossil forces you to work its way". Which is fine but you could... just decide to do it in git.
It does show that fossil is trying different approach, it doesn't want to be really distribute as much as just "synced"
https://www.fossil-scm.org/home/doc/trunk/www/fossil-v-git.w...
it's really too bad, everything else in Fossil looks very compelling.
If you are bound and determined to hide it by default, then, it's just a flag attached to a commit or a branch(by default in the CLI, it can only apply to branches):
fossil amend --hide <branchname>
fossil tag add --raw hidden <COMMIT_ID>
This is fossil version 2.20 [210e89a059] 2022-11-16 18:46:32 UTC
$ fossil timeline
=== 2022-12-13 ===
22:27:34 [dcb81ebb44] *CURRENT* paragraph (user: mdaniel tags: main, trunk)
22:27:09 [3a2c469dae] better readme (user: mdaniel tags: main, trunk)
22:26:46 [3fd12e216a] readme (user: mdaniel tags: main, trunk)
22:26:24 [222d705331] initial (user: mdaniel tags: main, trunk)
+++ no more data (4) +++
$ fossil tag add --raw hidden 3a2c469dae
$ fossil timeline
=== 2022-12-13 ===
22:30:39 [e54ead5c08] Edit [3a2c469dae6d6490|3a2c469dae]: Add "hidden". (user: mdaniel)
22:27:34 [dcb81ebb44] *CURRENT* paragraph (user: mdaniel tags: main, trunk)
22:27:09 [3a2c469dae] better readme (user: mdaniel tags: main, trunk)
22:26:46 [3fd12e216a] readme (user: mdaniel tags: main, trunk)
22:26:24 [222d705331] initial (user: mdaniel tags: main, trunk)
+++ no more data (5) +++
If you run `fossil timeline --sql`, you can then change the sql to skip the hidden flag, though I'm not sure where in the commit that is, one would have to do some digging.
Regardless, if you REALLY want to delete and/or hide history, one can do this in Fossil as well, it's a touch more annoying in Fossil compared to git, but it's not difficult or anything, you can do it all from the CLI. [This thread](https://fossil-scm.org/forum/forumpost/333117081652bf3db6dc6...) shows hiding branches and commits and [this stackexchange](https://stackoverflow.com/questions/42832992/shun-a-whole-co...) handles deleting data from the repo.
After all, SQL consists not only of the SELECT command; it has DELETE and INSERT too.
Also GH/GL is popular because of everything else they do (issues, page rendering, now CI/CD) not because git CLI is too hard.
Improvements are welcome of course but the moment newbie hits merge conflict no amount of CLI niceness can get around them not knowing what to do with it and that's true regardless of DVCS. For "non-programmers" no CLI is good enough anyway.
https://corecursive.com/066-sqlite-with-richard-hipp/
I’m going to write my own mail server. I was making notes on that even as we were setting up this call. That’s a big problem, and that’s at least as difficult if not more difficult than writing a database engine, but I don’t want to be beholden to Gmail. I don’t want them controlling my destiny. I don’t want them controlling the record of all of my conversations. I want to control that myself, and so I’m going to go through a lot of pain and a lot of work and a lot of effort to come up with some solution that I can control myself. I can go out and lease a virtual machine out there in the cloud and run it myself and not depend on a third party to control my email.
Wowzers, that's some RMS-"Web Over Email"-level "I have privacy concerns so I'm going to reinvent $foo" https://stallman.org/stallman-computing.html
Hipp not only wrote SQLite, which uses his own parser generator (Lemon) but he also wrote his own version control system (Fossil), and his own editor.
Perhaps it's more likely that he wants to be in control of the things which most closely affect him, rather than this being a privacy issue?
I've recently did a small server for private use (after google yeeted the free tier) and let's say it does require a bit of mail knowledge to set it up right.
That being said there is probably some container doing it, surely "easy mail selfhosting" isn't a new problem
Anyone had experience running SQLite on low-power embedded platforms? Is this expected performance?
sqlite3_initialize();
sqlite3 *db;
sqlite3_open("/data/sqlite.db", &db);
char *err = 0;
sqlite3_exec(&db, "DROP TABLE table_name;", NULL, NULL, & err); // This takes ~5ms even when table_name does not existThe handle named `db` in your example is as expensive to create as a connection in the relative sense and should be optimized the same way in your application. It may be less expensive than a network database connection, but its still very expensive relative to any subsequent queries. The database file is opened, its first page is read, its schema is parsed from text (!), some memory is pre-allocated and so on every time you make that call.
The sqlite3_exec call invokes the parser and query optimizer every time it is called. A better benchmark would be to compile a valid statement (sqlite3_prepare). The unit under test should just be sqlite_bind() for relevant parameters followed by sqlite_step(), and maybe (but not necessarily) sqlite_reset().
I wonder if it could be doing multiple small writes on every operation -- I could definitely imagine small writes getting very costly, especially with wear-leveled file systems that might be moving around pages on even small writes.
I'd expect closer to 5uS than 5mS, but I'm not that familiar with the ESP32 devices
Have you used a digital logic analyzer to figure out how long reading the page takes, how many pages SQLite reads for that query, and how much work SQLite has to do before it can fetch the next page? I haven't seen any ESP32 SQLite ports that can eagerly fetch pages, even though it's relatively simple to do with DMA (but would require some significant changes to SQLite because it's designed to be ignorant of OS preemption which is not the case in most RTOSes).
While I very much appreciate the safety of being a very small cog in a larger machine, I also wonder how my career might be more enjoyable and satisfying if I had chosen to limit my options to small teams where every single person on the team had a critical impact on the success of a project.
I think medium-sized projects (say a team of max 10, or maybe 2 smaller ones) seems to be the most fun ones. You have enough manpower to do some bigger reorganizing if required but it isn't as big that rewrite of any part is years-long saga.
SQLite is not something we wire up and expose bare-ass to the public internet. We don't worry about 0-day security issues in this context. It's more like a wrapper our product uses for structured disk access. Anything that would warrant urgent patching is in .NET vendor or in-house software piles.
If something horrible happened and the entire SQLite dev team was wiped off the face of the planet, I'd make a note to look at alternatives by ~2025.