Hacker News Reader

Comment by luhn | Hacker News Reader

The author seems to assume that everybody is using SQLite, but SQLite for a production database is an extremely niche choice. Attempting to expose more popular options like PostgreSQL or MySQL as SQLite would be extremely difficult because SQLite only supports a subset of SQL, whereas PostgreSQL and MySQL both implement their unique superset (for the most part) of SQL.

But it doesn't matter. The API doesn't matter. Web 3.0 was never about APIs, it was about data. A standardized API is only useful if it outputs standardized data. Having a bunch of bespoke SQLite tables scattered across the web gets us no closer to the ideal of Web 3.0.

View on HN

You may be underestimating SQLite here. It has grown a LOT of features over the past decade, many of which were directly inspired by PostgreSQL.

Features you may have missed:

- CTEs - the WITH statement works great. And recursive CTEs, which can do mandelbrot fractals! https://www.sqlite.org/lang_with.html

- Surprisingly good built-in full text search: https://www.sqlite.org/fts5.html

- Functions for directly querying JSON data in columns: https://www.sqlite.org/json1.html

- Extremely comprehensive geospatial capabilities thanks to the SpatiaLite extension - this has a huge amount of functionality, which I think is better than MySQL though not yet as good as PostGIS: https://www.gaia-gis.it/fossil/libspatialite/index

I'm not saying it's as "good" as PostgreSQL, but I don't thank your argument that PostgreSQL and MySQL implement a substantially larger portion of SQL holds up particularly well.

As luhn said, it’s more about a standard data format than a db choice. If every client has to figure out what schema a website uses for a recipe, let’s say, then Web 3.0 is still unrealistic.

Schema.org exists, but all websites adopting it seems unlikely.

That being said, I can maybe see a world in which one company adopts schema.org schemas and the rest have to follow suit to be competitive in that particular domain.

> Schema.org exists, but all websites adopting it seems unlikely.

Schema.org has the backing of major search engines and other reusers of Web-served content. It's way more likely to be adopted compared to anything else in its general domain.

SQLite is the most used database engine in the world, so I wouldn't call it niche. In fact, by some estimates, it is probably used more than all other database engines combined.

The only difference is that it is usually run locally (compared to Postgres and your other examples), but something doesn't have to run remotely to be considered running in production :)

Yes, when I said "production database" I meant a database for a web application. My iPhone running SQLite doesn't relate to Web 3.0.

> Yes, when I said "production database" I meant a database for a web application

I'm not sure that's what the author of the article means though, at least in my interpretation.

I assume when they say "everyone is already using it" I assumed that they meant literally everyone is using it on their phones and PCs every day, not that everyone is using it to develop production web applications (because very few people develop production web applications in the grand scheme of things!).

I presume they mean that it is one of (if not the) most common databases in existence in the wild, and it's interesting that it has this property of being able to be remotely read with surprisingly little overhead (without the need to implement an entirely bespoke database to be read in this way).

I'm not sure how to parse what the author is saying besides you should be exposing your database directly, which is apparently SQLite.

> In the process, it demonstrated a new kind of web app whose entire database was exposed and queryable from the outside.

> The data needs to be exposed in its original form; any additional translation step will ensure that most people won't bother.

I think you misunderstood the author.

"The data needs to be exposed in its original form; any additional translation step will ensure that most people won't bother. The beauty of this technique is that you are already using SQLite because it's such a powerful database; with no additional work, you can throw it on a static file server and others can easily query it over HTTP."

The author believes (IMO wrongly) that there's lots of web app data that can be exposed via SQLite-over-HTTP without translating it into SQLite, because it's already in SQLite.

The author is saying that since lots of web apps use SQLite for their production database, they can easily "throw" their SQLite DB onto the web. But, in that case, you're out of luck if you use Postgres, MySQL, Oracle, MS SQL Server, or any of the popular key-value datastores like Mongo, Redis, or Elasticsearch.

I think he does mean web apps/sites, at least in large part. He is talking about implementing of web 3.0, after all. OTOH, I suppose there's no reason why web has to apply primarily to things that are currently webstuff. You both make good points.

sqlite is actually quite robust in a production web application environment, I have used it as a database in several production applications over the years including one that serviced 600k MAUs without issue. if your application is very write heavy or you're FAANG scale it could present a problem, but IMO sqlite is probably the best bang for your buck solution for the workload of most websites and applications.

tbh I wish it were used more. it's much cheaper to run and just as fast as mysql (sometimes much faster) on your average wordpress blog or equivalent. you don't need to handle thousands of concurrent writes, only maybe 5 max at peak... and queueing them for tens of milliseconds is totally fine. as long as you're not writing horrifically inefficient insert operations, you absolutely won't notice until you're under ridiculously high load for most sites.

Yeah, I've heard the argument before that SQLite is just fine as a production database. Not arguing against that, just saying that it's not a common choice.

SQLite is not uncommon in web applications. It isn't just found in phone apps and OSs.

It works pretty well for tasks that are primarily read heavy and is a lot lighter doing so than the common non-nosql alternatives (SQL Server, postgres, mariadb/mysql, ...). For applications that need good performance in the presence of much writing, particularly where concurrent writes might be desirable, it is far less ideal though.

I think you can just just refer to "non-nosql" as... sql! :)

"nosql" tends to mean more than simply not SQL. Often not any sort of relational structure and related querying methods, for instance, so "not nosql" is not the same set of things as "SQL".

And yet notably all the examples you listed are sql...

And yet, you know that isn't the point and are just trying to have the last word.

SQL is not the only structured query language and SQL-queried databases are not the only relational databases. It just happens that the subset of database services that are not nosql and commonly “compete” with sqlite as a relational database mostly happen to use some variant for SQL.

This whole architecture flips a lot of assumptions on their heads. In particular here, the assumption that all user queries end up in the same database instance, which then might not handle the load, is thrown out. Queries here are primarily local to, and potentially defined by, the user. If that's useful to you, it's not something you'd want to use a traditional MySQL or Postgresql setup for.

Exposing you whole database (or subset user is allowed to see) as GraphQL is way better as it is engine agnostic.

GraphQL is not a standard, it's just a technology for building custom API's which are far from "agnostic" in practice. You can use SPARQL if interoperability if your goal.

SPARQL is 14 years old, and RDF it's supposed to query is 26 years old.

I think by now most people said "nah".

GraphQL is something people actually want to interact with and develop.

Both SPARQL and RDF are under continuing development, actually. Possibly more so than GraphQL itself.

If you have a Postgres database you can expose it as GraphQL by putting Hasura in front if it. A piece of software used by Walmart and Atlassian among others, despite being just 5 years old.

Nothing like that exists for SPARQL, RDF and Postgres

My point was not that people are using SQLite in prod everywhere; read that paragraph in more of a speculative voice, not a statement of fact about the present. At any rate, i do think the range request technique makes SQLite more practical to use in database-driven apps that normally would've opted for a traditional db like postgres (though there is more work to be done to make this technique fast when doing complex queries...lots of joins are no bueno right now).

I enjoyed the speculative bias there as its prompting me to pick up the sqlite bundle thats been sitting in my downloads while i use post-gres-like-the-rest

Well, having to write unique SQL per site is much better than having to write unique scrapers per site.

Exactly. And there’s really no reason those queries couldn’t be made public / collaboratively maintained.

You could probably take it one step further and define an OpenAPI spec which is populated via those queries. Tho that would require an intermediary / post-processing, likely with a cache.

Regardless, the capability to determine how and what to consume sits with the consumer (developer) from the outset. Rather than having to scrape the data, normalise it into some form of schema, and then build an api / interface around it. And then worry about keeping it up to date.

Wouldn't you want to use a subset of SQL in an API? Why use a unique superset that differs per webpage?

I thought Web 3.0 was about NFTs.

People are talking that way now. Before what is being called “Web3” ATM Web 3.0 was to be the “semantic web” (where Web 2.0 was the “interactive web” - with many things become read/write instead of read-only and greater interactivity, both in terms of social interactivity and individual interactive apps being web based, being the focus of technological enhancements).

This article is talking about that earlier definition, and a way it might once again be the definition, perhaps relegating Web3 to Web 4.0 (or web we-worked-out-it-was-a-ponzi-scheme-so-just-stopped with something else being Web 4.0, if you take the more cynical view).

"Before the term was hijacked by crypto-grifters (and, admittedly, a few genuinely neat projects), web 3 (point oh) referred to Tim Berners-Lee's project to promote a standard way to expose and parse metadata on the web."

Reply on news.ycombinator.com