SQLite's Use of Tcl (2017) | Hacker News Reader

SQLite's Use of Tcl (2017)

tcl-lang.org

113 pointsripe39 comments

I don't get the Tcl hate. I use it all the time on Cisco gear, and it's incredibly useful. Sure, if you try and turn it into a 10k+ LOC solution, life is going to suck. But in it's use case envelope, so much value.

But then I'm old and still use perl for small stuff, so probably not reading the room....

Time does move on, but not necessarily for good reasons. TCL is the best way to embed programmability into C or C++ code; Oousterhout’s writings on modularity and composability explain why this is so useful to those that lack the experience of winning with it. But we have to use YAML for ops instead and wait for the scarse and slow Go or Java or whatever teams to extend their yaml interpreters every time we need a value to be a loop instead of one value.

>TCL is the best way to embed programmability into C or C++ code

One of the best ways.

See also, Lua.

Last time I checked, embedding Lua in C or C++ was _way_ easier than embedding Tcl

Could you please unpack that? I'm really curious what the differences are.

I am 31 years old and I love Tcl and Perl, and I started my programming journey with C at age 13-14, so I am not sure how old you are! :P

> To help the team stay in touch, a custom chatroom has been created using a Tcl/Tk script. The same script works as both client and server. The chatroom is private and uses a proprietary protocol, so that developers are free to discuss sensitive matters without fear of eavesdropping. The chatroom is implemented as just over 1000 lines of Tk code, and is thus accessible and easy to customize.

Curious if anyone has more details on this. Does it have encryption?

The paper is from 2017. Fossil got chat support in 2021 and the developers now use Fossil-chat. https://fossil-scm.org/home/doc/trunk/www/chat.md

Fossil chat has the advantages that (1) it is fully encrypted and (2) it works from any web-browser, including on mobile phones.

Does fossil have something similar to Git-LFS? I'd like to store binary assets like PNGs and music files and such

It by default doesn't allow any binaries at all, nor CRLF files <https://fossil-scm.org/home/help?cmd=commit#:~:text=may%20be...>, nor whatever default value it has for "oversized"

That said, to the best of my knowledge git-lfs operates upon stdin and stdout, like much of git, so I'd guess you could actually just commit the tracking file and manually run $(git-lfs scrub) et al. I do hear that "manually run" isn't the same as the way it works in git, but that's why fossil does things the fossil way

Incorrect. The whole sentence is:

"may be aborted if a file contains content that appears to be binary, Unicode text, or text with CR/LF line endings unless the interactive user chooses to proceed. If there is no interactive user or these warnings should be skipped for some other reason, the --no-warnings option may be used."

I use fossil and checking in binaries works beautifully. You _do_ get a warning, but as seen in the documentation, you can use --no-warnings if you don't want that.

Note however, that you can't use diff on binaries, and since the entire history of the repository and the versions is shipped to all developers, storing large binaries quickly becomes cumbersome. I would in that case, store links to binaries, which themselves are stored in an archive, or switch to another scm program.

    Note however, that you can't use diff on binaries, [...]

You absolutely can, but you need to use an external diff tool:

   fossil diff --command "compare"

You can also customise the diff-command variable. You might need to pass --diff-binary .. I forgot.

As for storing binaries, unversioned files have no history, and are not synced automatically.

Not the same thing, but in addition to binary file support as mentioned in other comment, fossil supports adding unversioned files to the repository that are not version-managed at all. Might make sense for some large files if you just want it available but not versioned (only using space for the latest version) or automatically checked out.

https://fossil-scm.org/home/doc/trunk/www/unvers.wiki

Yes, it is called "unversioned files" in Fossil, but the max size is limited to what SQLite can handle in a single blob (so ~4Gbytes)

https://fossil-scm.org/home/doc/trunk/www/unvers.wiki

> On the server-side, message text is stored exactly as entered by the users

I suppose the encryption is only at the TLS layer?

> I suppose the encryption is only at the TLS layer?

Correct unless the fossil repository in question uses SQLite's SEE (encryption) extension (which fossil can, but relatively few repositories use that, AFAIK).

Fossil comes with a chatroom feature (https://fossil-scm.org/home/doc/trunk/www/chat.md). Could that be what they're referring to?

they refer to it's precursor, as per a sibling comment.

E2EE group chat in 1000 lines would be rather impressive

With a lot of the code that may be stashed away into libraries, it doesn't seem all that remarkable. I think the higher level logic and control flow for E2EE group chat could be condensed to 1000 lines with the proper abstraction. Tcl probably helped with that abstraction so credit where it is due.

Fossil-SCMs chat is not E2E encrypted but it does at least use TLS.

relevant links:

https://en.wikipedia.org/wiki/Modified_condition/decision_co...

https://shemesh.larc.nasa.gov/fm/papers/Hayhurst-2001-tm2108... (This tutorial provides a practical approach to assessing modified condition/decision coverage (MC/DC) for aviation software products that must comply with regulatory guidance for DO-178B level A software)

> It turns out that sqlite3_analyzer, though disguised as an ordinary executable, is really a TCL application. The main source code file for this application is tool/spaceanal.tcl. During the build process, this script is converted into a C-language string constant (using another TCL script) and added to a very simple C-language wrapper than starts a TCL interpreter and then passes the application script to that interpreter.

Haha, didn't know that. That's cool.

While most of this looks cool, the stuff about parsing vdbe.c's switch cases and assigning opcodes seems a little too much for my taste.

> SQLite supports this syntax. But because of its TCL heritage, SQLite also allows the parameter to take the form of a TCL variable. Hence:

    SELECT passwd, photo FROM user WHERE uid=$uid

Did they put "eval" in SQL parameter processing? Is there an SQL injection attack vulnerability there?

> Is there an SQL injection attack vulnerability there?

No, at least not if you put the SQL inside of {...}, which IIRC the documentation strongly recommends.

The $uid is passed down into SQLite. It is a single token recognized by the SQL parser itself. It does not get expanded by TCL. The $uid token serves the same roll as a "?" or ":abc" token would in some other SQL implementations. It is a placeholder for a value. The tclsqlite3.c interface first parses the SQL, then asks for the names of all of the placeholder tokens. Then it binds the values in TCL variables of the same name to those placeholders.

Indeed, this whole mechanism is specifically designed to make it easy to write SQL-injection-free code. As long as you put your SQL inside of {...}, you are completely safe from SQL injections.

If your TCL script includes SQL text inside of "...", then TCL will do the expansion and SQL injection is possible. But as long as the SQL text is inside of {...}, SQL injection is not possible.

Fully cognizant that I'm rolling the dice by responding to this comment, but isn't picking a variable syntax that could resolve in the unsafe way a "you're holding it wrong" waiting to happen?

  % set ex1 "SELECT * FROM FOO WHERE alpha=$bravo"
  can't read "bravo": no such variable
  % set ex2 "SELECT * FROM FOO WHERE alpha=?1"
  SELECT * FROM FOO WHERE alpha=?1

Don't get me wrong, https://peps.python.org/pep-0249/#paramstyle allowing %s and %(alpha)s are similar footguns and I wish they didn't exist, but at least they don't automatically resolve in the way that $ does in Tcl

It's an idiomatic Tcl thing. E.g. `expr` (the standard word used to evaluate infix expressions like `1+2`) does the same exact thing, handling the expansion itself and expecting the caller to use {} to ensure that a variable expansion not incorrectly treated as a bunch of operators. Similarly, when you're writing the condition for a loop, you need to use {} to delay expansion so that the loop word can do it anew for each iteration. One can argue that this is somewhat error prone, but at least there's a consistent pattern here, and once you know what it is and why it exists, it's pretty straightforward.

Reply on news.ycombinator.com