HNHacker News
TopNewBestAskShowJobs

SQLite

2,616 karma · joined October 16, 2012

Dr. D. Richard Hipp is the creator of SQLite and the Fossil DVCS. He lives in Charlotte, NC.
submissionscomments
SQLite··on Ask HN: Adding non-commercial clause to open source license?
> Open Source Definition explicitly excludes [public domain]

Mis-information. The Open Source Definition (https://opensource.org/osd) has no such exclusion. Read it for yourself.

SQLite··on SQLite 3.38 Released
I like anamexis's explanation better than my own (that I was typing in concurrently). :-) I'm going to upvote his...
SQLite··on SQLite 3.38 Released
No date/time data type will meet everybody's requirements. Some date/time formats will have a limited range, or won't have enough precision, and on the other hand, some date/time type implementations will give lots of range and precision but then end up taking up too much space on disk. It is not possible to please everyone. It seems better to provide basic low-level datatypes (integer, 64-bit IEEE float, string, BLOB) and let the developer choose whatever date/time representation best meets the needs of the application.
SQLite··on SQLite Release 3.37.0
The ANY datatype in STRICT mode behaves the same as a column with no datatype at all in ordinary SQLite. ANY does not give you any new capabilities.
SQLite··on SQLite Release 3.37.0
The SQLite dev's internal performance testing using cachegrind shows that STRICT mode uses about 0.34% more CPU cycles. So STRICT is slightly slower, but not enough that you could measure the difference reliably in a real-world system.
SQLite··on Why SQLite does not use Git (2018)
The incorrect merge is preserved because that is what actually happened. The incorrect merge was published. People saw it (and commented on it in the SQLite Forum). If I "disappear" the merge, that would be airbrushing history. The correct solution is to fix the problem, while maintaining an immutable audit trail, not to delete the problem.

When I was in high school, I was taught that if I worked as a bookkeeper and I make a mistake, I should never erase the mistake. Instead, draw a line through the mistake, notate what is wrong, and enter a correction. To erase an entry in the financial ledger of a company is fraud. It is a felony. Making a correction is fine. But do not erase. Always preserve an audit trail.

I believe that VCSes should be treated similarly. While you are assembling a change, you can make as many erasures and corrections as you like. But once you commit the transaction - once you check-in the change - it then becomes part of the permanent record. To alter that transaction after the fact is akin to felony fraud. Sure, mistakes happen. By all means, correct the mistakes. But the original mistake and the correction should all be part of the audit history.

If you want to say that commits to your private branches are not part of the permanent record, and that you should therefore be permitted to edit those private branches, then I think you have a stronger case. That does not come up as much in Fossil. Fossil does support private branches, but they are seldom used. The usual case in Fossil is that all check-ins auto-sync up to the parent repo.

Shunning is not quite the same. Shunning is a mechanism for removing illegal are illegitimate content. Shunning is sometimes required to comply with legal mandates. But it is not a part of day-to-day practice. Shunning is an exception - and escape valve - undertaken only in an emergency.

SQLite··on Why SQLite does not use Git (2018)
Fossil does not have a "rebase" command, but anything you can do with rebase you can do in Fossil, and more.

Fossil allows you (for example) to fix typos in the commit messages of published branches, safely and in a way that does not destroy history and propagates cleanly with "sync". It allows you to modify the DAG safely, and in a way that does not destroy history and propagates cleanly. It does this by providing special tags that when added to a commit change the check-in comment, or branch name, or parents of the commit. The original immutable check-in is preserved, but for display purposes, the tags can override some properties of the original check-in.

Real example from just this morning: Last last night I mistakenly merged the wrong way. I merged the reuse-schema branch into trunk, rather than merging trunk into the reuse-schema branch. When I saw the problem this morning, I was able to fix it, even though this mistake had already propagated to other repositories. The change entered the DAG as a supplemental "correction" tag, so no history was lost, and if in 20 years somebody wants to go back and figure out what happened there, they can, because all information is preserved. But for day-to-day viewing of history, it looks as if I had done the merge correctly to begin with.

SQLite··on Strict Tables – Column type constraints in SQLite - Draft
coincidence
SQLite··on Strict Tables – Column type constraints in SQLite - Draft
Scan this HN thread to see comments from devs who are aghast. :-)
SQLite··on Strict Tables – Column type constraints in SQLite - Draft
In something like this:

CREATE TABLE t1(a INT, b TEXT); INSERT INTO t1(a,b) VALUES(1,'2'); SELECT * FROM t1 WHERE a=?;

The type of the ? is ambiguous. You can say that it "prefers" an integer, but most RDBMSes will also accept a string literal in place of the ?:

SELECT * FROM t1 WHERE a='1'; -- works in PG, MySQL, SQLServer, and Oracle

SQLite··on SQLite: Vulnerabilities
Complete history: https://www.sqlite.org/docsrc/finfo/pages/security.in

The title was updated today in response to one of the posts above.

SQLite··on Git vs. Fossil: what you should have done vs. what you did
We accept (quality) patches and contributions from people who have completed the necessary paperwork to put their patches/contributions in the public domain. This restriction is in place so that that SQLite itself can remain in the public domain.

If we accept drive-by patches from anonymous users, then SQLite would have to be relicensed as GPL or similar, which would undermine many of its use cases.

SQLite··on Git vs. Fossil: what you should have done vs. what you did
See https://www.sqlite.org/src/reports?type=ci&view=byuser

SQLite has had about 33 contributors (after you combine people using multiple login names). But 92% of the commits have been from just two people. 97% from just three people. Then there is a long tail of other contributors.

Fossil itself is similar: https://fossil-scm.org/home/reports?type=ci&view=byuser - 5 or 10 people account for most of the commits, and then there is a long tail.

But aren't most projects like this? A few core developers are responsible for most changes and enhancements, and then there lots of others that might contribute a patch or two here and there? I suspect there exceptions to this (the Linux kernel comes to mind) but I think they are rare. Correct me if I'm wrong.

SQLite··on Git vs. Fossil: what you should have done vs. what you did
> [Fossil is] basically incapable of rewriting your development branch into something more logical...

False. All you need to do is start a separate "presentation branch" and cherrypick and/or merge changes from your "development branch" in as necessary to make it logical for review. Not difficult to do.

The difference from Git is that in Git you end up discarding the development branch and keeping only the presentation branch, whereas Fossil preserves them both.

Fossil does anything Git will do, except for one thing: Fossil does not (easily) sync individual branches. (You can do it, but it is a pain.) Git comes with the idea that individual developers keep their own private branches and only share them after they've been cleaned up. The idea behind Fossil is that everybody shares everything.

By analogy: Git is like where everybody has their own office with doors that close, and developers emerge from their own office from time to time to share their work or collaborate. Fossil is more like an open office with no walls or doors.

There are advantages and disadvantages to both approaches. I encourage you to use whichever one works best for you. If you like the Git approach, then by all means use Git. I find that the Fossil approach works better on the projects that I manage, but that just me. You do whatever works best for you.

Bottom line: Fossil was created to support SQLite development. It does this very, very well. If it never does anything else, it will have been a great success. Any use of Fossil beyond SQLite is just gravy. As it turns out, many other developers have found Fossil useful, too. But perhaps your usage patterns are different and the Fossil model does not work for you. That does not invalidate the fact that it works well for others.

All that said, there are many ideas in Fossil that could be copied into Git without changing the Git development model. So even if you don't like Fossil's development model, you should still look at Fossil, so that you can steal ideas (and/or source code) to import into Git and hence make Git better.

SQLite··on The Untold Story of SQLite
> would sqlite really be worse today if you hadn't done your own VCS.

Yes. Fossil is not only the VCS for SQLite, Fossil is also built around SQLite. SQLite is a core component of Fossil. Thus, when I am working on Fossil, I am forced to interact with SQLite as a "user" instead of as a "developer". In geek-speak, it forces me to "eat my own dog food". This, in turn, prompts me to add needed features to SQLite and more generally to make the SQLite interfaces friendlier to application-developers.

One recent example: SQLite version 3.34.0 added the ability to include two or more recursive terms in a Recursive Common Table Expression. (See item 2 in https://www.sqlite.org/releaselog/3_34_0.html and subsequent links.) This feature was added specifically so that I could more easily write SQL statements that would walk the Fossil version history DAG, as described by the https://www.sqlite.org/lang_with.html#rcex3 link.

I did not develop Fossil with this "dogfooding" idea in mind. It was an unanticipated benefit of Fossil. But in the end, I think it might have been the most important benefit of using Fossil instead of some other VCS.

SQLite··on The Untold Story of SQLite with Richard Hipp
Maybe you could steal ideas and/or code from Fossil to add to Git in order to make Git better?
SQLite··on Wapp – A Web-Application Framework for Tcl
I am the developer of Wapp. I gather (from other comments) that the name "Wapp" collides with a recent pop song by someone named "Cardi B" - please correct me if I have misunderstood. For the record:

  *  I do not follow or participate in pop culture.
  *  I do not listen to pop music, ever.
  *  I do not know who Cardi B is.
Any similarity in the name of the Wapp software framework and music by the individual or group known as "Cardi B" is purely coincidental.
SQLite··on Althttpd: Simple webserver in a single C file
Linode doesn't follow the cafeteria pricing style. You buy a package. $40/month is the minimum for us to get the disk space and I/O bandwidth we need. We could get by with less CPU, perhaps, but the extra memory and extra cores do reduce latency and they are nice to have on days when SQLite is a top story at HN. (Load avg has been running at about 0.95% all day today.)
SQLite··on Althttpd: Simple webserver in a single C file
Look again. The entire Althttpd website is 100% dynamic. Notice that the hyperlink at the very top of this HN article is to a Markdown file (althttpd.md). A CGI runs to convert this into HTML for your web-browser.

The core SQLite website has a lot of static content, but there are dynamic elements, such as Search (https://www.sqlite.org/search?s=d&q=sqlite) and the source code repository (https://www.sqlite.org/src/timeline?n=100&y=ci).

So far today, 23.48% of HTTP requests to the sqlite.org domain are for dynamic content, according to server logs.

SQLite··on Althttpd: Simple webserver in a single C file
The "z" prefix is intended to denote a "zero-terminated string", or more specifically a pointer to a zero-terminated string.
SQLite··on Althttpd: Simple webserver in a single C file
Fossil does has the ability to squash commits.

The difference is that Fossil does not promote the use of commit-squashing. While it can be done, it takes a little work and knowledge of the system. Consider the premature-merge problem in which a feature branch is merged into trunk before it is ready, and subsequent typo fixes need to be added. To do this in Fossil you first move the errant merge onto a new branch (accomplished by adding a tag to the merge check-in) then fix the typo on the original feature branch, then merge again. So in Fossil it is a multi-step process. Fossil does not have a "rebase" command to do all that in one convenient step. Also, Fossil preserves the original errant check-in on the error branch, rather than just "disappearing" the check-in as Git tends to do.

The difference here is a question of priorities. What is more important to you, an accurate history or a clean history that tells a story? Fossil prioritizes truth over beauty. If you prefer a retouched or "photoshopped" history over an auditable record of what really happened, Fossil might not be the right choice for you.

To put it another way, Fossil can squash commits, but another system might work better for you if commit-squashing is your go-to method of dealing with configuration management problems.

SQLite··on Althttpd: Simple webserver in a single C file
Indeed. And a Citation biz-jet is way faster, flies higher, goes further, and carries more passengers than a Carbon Cub. On the other hand, the Citation costs more, burns more gas, takes more maintenance, and is more complex to fly, and you should not try to land a Citation on a sandbar in a remote Alaskan river.

Choose the right tool for the job.

Changing the https://sqlite.org/ website to run off of Nginx or Apache instead of alhttpd would just increase the time I spend on administration and configuration auditing.

SQLite··on SQLite is not a toy database
See https://www.sqlite.org/fts3.html#custom_application_defined_...

Since 2016, arguments to the fts3_tokenizer() function must be variables (ex: ? or :var or @var or $var) to which values are supplied by the application at run-time using sqlite3_bind_pointer(). There is no way to do this from pure SQL script. Nor is there any way to do this from within a view or trigger. There is no way to invoke fts3_tokenizer() from a maliciously corrupted schema or database.

SQLite··on SQLite is not a toy database
SQLite requires that the self-reference be in the top-level FROM clause of the recursive part of a recursive CTE. PG apparently allows the self-reference to be down inside of subqueries, as long as there is only one reference.

I have make a copy of the Collatz Conjecture CTE that you linked to and was going to see if I could get it to work in SQLite for the next release cycle. I don't (yet) see any reason why it shouldn't work to have the recursive reference down inside a subquery, as long as there is only one recursive reference. No promises. We'll see how it goes.

SQLite··on Fossil Chat
The language in the rebase article [1] has been softened somewhat.

[1]: https://fossil-scm.org/home/doc/trunk/www/rebaseharm.md

SQLite··on Fossil Chat
I consider a version control system to be a history of a project. If you change the history to something that is materially different, which rebase does, then you are lying about the history. You cannot white-wash this fact.

The history Git and in Fossil is only precise to the transaction level. A key-stroke or backspace is not a transaction. A single iteration of the edit-compile-test cycle is not a transaction. A transaction is created by the "commit" command. Git and Fossil make no record of the stuff that happens in between two commits. They only record what is in each commit. So you can backspace and edit and change all you want to before typing "commit". But once you type "commit", all that you've done since the previous commit becomes part of the permanent record. Rebase violates that constraint. It changes the permanent record. Rebase makes the repository say that the sequence of commits is different from the order in which they actually happened.

You can call that whatever you like. "Telling a compelling story." "Generating a clean history." I call it "lying", since the purpose seems to be to deceive the reader about what actually happened.

It is useful to have a simplified overview of project history, to aid the reader's comprehension. There is no sin in this as long as you are truthful to state that the simplified history is in fact a simplification that skips or revises some of the messy details of reality. The alternatives to rebase that Fossil provides do exactly that - they suppress the unimportant details of the history to provide a simplified and more readable history. The difference is that the original, truthful history is still provided, for auditing and for those who care. In other words, the repository reports both "what we should have done" and/or "the best way to think about what we did" in addition to "what actually happened".

I concede that if you think of a source code repository as just a story or as documentation to help future readers, and not as a accurate history of the project, then rebase is not lying. In that case, rebase is just revising your narrative. But I think that a source code repository should be a true history of a project. If you view a source repository as a true history, as I do, then rebase is a tool for lying about the history.

SQLite··on SQLite is not a toy database
This is misinformation. SQLite does not execute arbitrary code found in the data file. There was a bug, long since fixed, that could be used by an attacker to cause arbitrary code execution upon opening the database file. The referenced video talks about it. It was a very clever attack. But the bug that enabled the attack was fixed even before the talk shown in the video was given.

Let me say that again: SQLite does NOT execute arbitrary code that it finds in the database file. To suggestion that it does is nonsense.

See https://www.sqlite.org/security.html for additional discussion of security precautions you can take when using SQLite with potentially hostile files. The latest SQLite's should be safe right out of the box, without having to do anything mentioned on that page. But defense in depth never hurts.

SQLite··on Fossil Chat
Newly registered users still don't have access to chat. I think if you add the "C" privilege to the generic "reader" user, that will solve the problem.
SQLite··on Fossil Chat
It's a fair question. I need to better document the security features built into Fossil. Here is a quick off-the-top-of-my-head summary:

(1) Fossil is designed to run inside a minimal chroot jail. The stand-alone "fossil" binary needs its repository database file, /dev/null, and /dev/random and nothing else. It can run inside a sparse jail. So even if somebody manages to find an RCE in Fossil, the damage potential is limited. They cannot shell-out because there is no /bin/sh in the jail.

(2) Fossil processes each HTTP request in a separate process.

(3) Custom static-analysis tools run over the Fossil sources at compile-time, and abort the built if any problems are seen. The source code and docs to these tools is in the Fossil source tree.

SQLite··on Fossil Chat
Thanks for the demo server, exikyut.

Maybe: Go to the Setup/Access page and select the "Allow users to register themselves" checkbox, and add permission "C" to "Default privileges". Then press Apply. With that setup change, users will have a "Create New Account" button on the login page.

← PreviousPage 3 of 8Next →