Hacker News Reader

Comment by vaughan | Hacker News Reader

I hate the way so much of programming is just fiddling knobs on some old systems. It always gets to a point where I would prefer to rewrite the whole thing myself with only the functionality I need and to truly understand what is going on under the hood. I just wish these projects were built in a modular way so you can re-assemble parts of them back together easily.

View on HN

That's the eternal pendulum of programming:

1. Ugh this thing is too integrated, we should break it up into modular parts so we can re-assemble them using only the parts we need. (Examples: most OS kernels, sqlite apparently, big excel spreadsheets, web frameworks like Rails)

2. Ugh this thing is too modular, you need a bazillion parts just to get a useful system. (Examples: analytics platforms consisting of 20+ unix tools strapped together in a pipeline, most microservices architectures, everything in the node ecosystem)

Spot on.

SQLite is a really remarkable piece of software -- decades of peak human capital have been invested in it. Why wouldn't you want to make the most of that to achieve your objectives?

> decades of peak human capital have been invested in it

Measuring software in human capital isn't a good idea. Software becomes crusty very quickly. Once you get a huge user base re-architecting becomes almost impossible without a re-write.

A lot of velocity can come from completely freeing yourself from an existing solution and it's customers in the short-term.

SQLite’s source code is well structured and pretty clean. It’s often held up as an example of a big codebase that’s great to learn from.

Google didnt, thus Chrome started replacing sqlite with https://github.com/google/leveldb

if i was an sqlite user, i really wouldn't care at all about google switching from sqlite to that thing in chrome.

From the readme:

> LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

it's a completely different thing from sqlite. sqlite is a relational database (which can also do key-value storing, I guess).

my educated guess is that google started with sqlite, then reached the peak usefulness *within chrome* and only then switched to developing its own custom solution.

if anything, this is a testament to sqlite and how far you can go with it before needing a custom solution.

Thats nice and all, but Im a Chromium user who wants to manually audit and edit my \User Data\Default\Local Storage. That wasnt a problem in sqlite days, tons of tools available. Switch to leveldb not only made that difficult (one paid GUI tool in existence afaik), but also introduced a change in localstorage behavior. Chrome localStorage used to return entries sorted by key, this is no longer the case.

I believe Chrome has deprecated LevelDB at this point. It turns out that LevelDB doesn't work well when you need many different databases in a single application. So all new Chrome storage is SQLite. At least until they find something better...

Whatever, blame it on chrome/chromium.

We’re talking sqlite here, your comment is off-topic.

A key-value store can underpin a SQL layer. Look at CockroachDB's usage of RocksDB (which is a descendant of LevelDB). Seems like a fair comparison to compare BTrees and LSM approaches. The latter work better because they align more closely with how solid state storage works. BTrees require a fair bit of pointer chasing for reads and are expensive to modify.

A product written by some notorious rock stars at Google, and with a reputation for corrupting data...

[1] - "...LevelDB has history of database corruption bugs.[15][16][17][18][19][20] A study from 2014 has found that, on older (non-checksummed) file systems, the database could become corrupted after a crash or power failure.[21]..."

https://github.com/google/leveldb/issues?utf8=%E2%9C%93&q=co...

https://bugs.chromium.org/p/chromium/issues/detail?id=261623

https://forum.syncthing.net/t/panic-leveldb-table-corruption...

https://github.com/google/leveldb/issues/333

https://ethereum.stackexchange.com/questions/1159/corruption...

[1] - https://en.wikipedia.org/wiki/LevelDB

It's worth pointing out that once you reach Google size and domination you can pretty much do whatever you want*. Things that if done by "ordinary" companies would destroy them or wipe out their customers/users overnight.

* Some limitations still apply.

You seem to believe a description like that is a recommendation, but it might be the opposite. Any corporation is a place where Not-Invented-Here virus spreads rapidly, because a) there are numerous problems and edge cases that develop at large scale; b) there is manpower to charge at them; c) there are incentives to do just so, and appear as definitely hard working to the management. That aura even spreads externally, in particular, to this conversation we have. If something is made by a hip company, it's a gods' gift to mortals, if it's from uncool place, it's definitely a rusty tech working on punch cards. (I am sure there exist at least one example in which the same person or team did both.)

Of course, they must have had valid reasons to use LevelDB for some components in Chrome. It seems that Minecraft-like world generating games are also a good use case for it. What I'm saying is that Google doing something does not automatically mean you need to do the same at all.

I think their reason was yolo. Cookies are still stored in sqlite so now they have two storage engines doing same thing to worry about.

I definitely feel you. What's interesting is that it seems like a lot of the big popular tools everyone uses all started out this way. Someone thought "man, all the existing solutions suck and I don't understand them - let's make a new one that does only what I need." Then it turns out, many other people also needed that. Then people who need slightly different things come in and knobs are added.

FWIW, the whole reason for SQLite is the observation that getting storage correct was extremely hard. Even with a filesystem and knowing to do fsync+rename, you can have data loss in surprising scenarios. The defaults for SQLite are tuned for durability and even still perform reasonably well. A lot of the tuning is about turning off that durability default.

These knobs by the way are part and parcel for generic storage systems. Including filesystems a bit too and definitely all databases I've ever seen (because different HW will have different optimal settings).

> I hate the way so much of programming is just fiddling knobs on some old systems.

actually, i believe it's quite the contrary: most of what we need nowadays it's already been invented and developed.

you can be incredibly successful as an engineer at many companies just by knowing what knobs are there and how to turn them properly.

and i think this is relevant because most people end up reinventing a square wheel that only does 10% of what the most used, widely spread, already-open-source wheel does.

To be honest, the author is literally fiddling knobs to enable existing features described in documentation and mentioned everywhere on the net. All the “performance tuning” here is done by SQLite developers.

SQLite is so modular that someone could write a replacement for the filesystem layer that ended up sending requests across HTTP to query a database on another server [1]. Without touching any other layer of the code. How much more modular do you want a database to be, without making other compromises?

[1]: https://phiresky.github.io/blog/2021/hosting-sqlite-database...

hmmm, running the dom demos causes a segfault in the browser. That probably shouldn't happen...

Reply on news.ycombinator.com