Hacker News Reader

Comment by xnorswap | Hacker News Reader

I don't understand these "DB in browser" products.

If the data "belongs" to the server, why not send the query to the server and run it there?

If the data "belongs" on the client, why have it in database form, particularly a "data-lake" structured db, at all?

A lot of the benefits of such databases are their ability to optimise queries for improving performance in a context where the data can't fit in memory (and possibly not even on single disks/machines), as well as additional durability and atomicity improvements. If the data is small enough to be reasonable to send to a client, then it's small enough to fit in memory, which means it'll be fast to query no matter how you go about it.

The page says one advantage is "Ad-hoc queries on data lakes", but isn't that possible with the most basic form that simply sends a query to the database?

What am I failing to understand about this category of products?

View on HN

I think a large benefit to "DB in browser" is about the whole "local-first" software movement. A good overview is written here https://sqlsync.dev/posts/stop-building-databases/

The essence of the approach is that a large majority of FE apps have constructed quite complex caching layers to improve performance over "querying backend data" - take a look at things like Next.js or React Query <- as the post above mentions, they're essentially rebuilding databases. So instead this approach just moves the db to the browser, along with a powerful syncing layer.

I think it's an approach that deserves more attention, especially to improve DX where we end up writing a whole lot of database-related logic on clients. Mind as well then just use a database on the client as well

To the other good points, I'd add that when people start making queries that aren't O(1) or O(n), modern computers and networks are capable of reasonably conveniently moving around amounts of data where it becomes practical for remote client to use its rather substantial power to answer questions on a gigabyte or two of data with its own computing power, and if enough people are doing this at once the combined client power they can have can easily overpower any reasonable amount of cloud compute you might be willing to deploy for this problem as they run non-trivial queries. I may be happy for dozens of clients to download some data and chew through some O(n log n) with mixed O(n^2) components on their own whereas I would not care to provision enough cloud compute to handle it all.

I think people forget that as cheap as cloud compute is, client compute is even cheaper. Generally it's already paid for. (Marginal electricity costs are lost in the noise of everything else you're supporting a client user with.) And while the years of doubling every 1.5 years may be gone, clients do continue to speed up, and they are especially speeding up in ways that databases can take advantage of (more cores, more RAM, more CPU cache). Moving compute to clients can be a valuable thing in some circumstances on its own.

This exact argument has been made since people had PCs on their desks (the 1990s, at least). We're still trying to figure out how to do it well.

Yes, but as the numbers advance from "the remote desktop has like 128KB" and "the remote desktop can chew through gigabytes without much stress", the delta between O(n) and O(n^2) opens up a lot more. It is perhaps a bit counterintuitive, but as systems grow in capability that delta grows.

Server capacity has grown along with it if not even faster.

A server in 1990 had total storage and RAM roughly comparable to a low-end mobile phone today.

Server capacities have not grown at O(n^2) on data sets.

Computational complexity classes are not intuitive things. You can't just go "oh, that's much bigger than that so it must that many times better".

> If the data "belongs" to the server, why not send the query to the server and run it there?

We could, but if the data size is not that huge, sending it once to the client and then letting the client perform the queries can be desirable. The tool works without internet access, the latent is much better, all results are coherent etc

> If the data "belongs" on the client, why have it in database form, particularly a "data-lake" structured db, at all?

Just because it fits in memory doesn't mean that the shape of the data does not matter. A data structure optimised for whatever query/analysis you want to perform has a significant impact on how fast and efficiently you can perform those operations

This argument boils down to:

- database engines should always run on a server, never on the user’s own machine.

- local data is never larger than the database engine, and it’s better in all cases to move all of the local data to the server where the database engine runs.

- local data can always be moved to another machine, regardless of sensitivity.

Buy are these always true, all the time? It seems to me that there are many use cases (e.g. training local AI models) where these rules should not be so absolute.

There is an increasing subset of people (think those that used to work in MS Access, Excel power users) who learned SQL in a business IT course or on the job. They don’t have data sized to fit in a DB, but they do want to do analyses that use window functions and things that SQL makes more natural than Excel vtables or functions. They may have to give reports to an equally technical boss who would like to play with the report and explore assumptions using SQL.

The data size is typically not large; SQL and integrating with cloud-based spreadsheets is the selling point.

Okay, so it's more about SQL than the database aspect?

I can see there'd be demand for that, but I'm not convinced the porting a full-scale DB to WASM is the best way to achieve that goal.

Once you accept the idea that you're going to run SQL on the browser, the forces are going to inevitably pull you to having an entire DB engine in the browser anyhow. You're going to need the entire query parser, enough of the IO code to be able to read the DB (which is a lot of it), you're going to want the optimizer, you're going to need RPC to send the SQL and read the results... yes, there is certainly some stuff you can trim, but it's not like it's a horrible idea.

So even if someone, say, started a startup with the idea "I'm going to present an MVP of an in-browser DB engine people can use", in 5-10 years it'd be a nearly fully-fledged DB anyhow.

Moreover, DuckDB isn't exactly a “full-scale” DB in the sense that someone would think about Postgres/MySQL. It is an embedded database more like sqlite that happens to be good for analytical workloads. It doesn't have to worry about things like HA, user auth, network wire formats, etc. that a usual “full-scale” DB would.

There are also some data lakes (though I have not really worked with any) that are homegrown and consist of smallish parquet files in a blob store; they might have several databases looking at them at any given time, and an embedded browser DB provides a frontend that is easier to stand up and auth against in some ways.

There may not be a reporting DB available to send queries to, in the traditional sense.

I don't know about DuckDB, but SQLite's Wasm build is well under 1 MiB minified/gzipped.

There's a lot that could be trimmed, but at that point, why?

I built a web application that became a progressive web app because it needed to work offline because it was used by a pilot during their work, and being at high altitudes, internet is not guaranteed. We cached a bit on localStorage, and I'd loop trying to make HEAD requests (there was no "am I online" API available...) and once we knew you were online we would push the locally cached changes up to the DB. In the meantime we still displayed the current data as you've changed it as well. It worked out nicely.

There's a lot of static data that never changes, that if you can request once, and cache it somewhere, it becomes really useful. It's less requests coming to the server. Heck, Tumblr in the 2010s would display a lot of blog data on the dashboard such as follower count, likes count, etc they had outages all the time, when I started to see less and less outages was when they stopped showing all that blog metadata on the dashboard because you didnt have hundreds of thousands (millions?) of users refreshing to see new content, querying across dozens of databases to see likes, follows, etc. It was very common to frequently refresh tumblr to see the latest content.

Imagine had they cached this data instead and only updated it based on a lastUpdated fields value being different.

I think the "db in browser" idea is interesting because it makes the user pay for the compute for many more of the typical DB tasks.

Any time you're in a situation where you're in an aggregation point you need to be wary of Moore's law; this is load balancers, database systems, log aggregation, etc.

If your computers are getting N cheaper / faster per year, but you're getting M new clients and all your clients are also getting N cheaper / faster per year, you're going to be getting more traffic faster than you can scale, unless you figure out how to cheat. Cheating may be sharding your internal workloads, but it may also be "make someone else do the work."

That analytics in the browser is about 10000x times more performant, and doesn't contest on a shared resource.

Right - so not a database, but a columnar analytics compute engine. Half of why we wrote arrow js was for pumping server arrow data to webgl, and the other half for powering arrow columnar compute libraries & packages like this.

For sub-100ms smooth interactivity, for data in a certain size range sweet spot, can be very nice!

Reply on news.ycombinator.com