Hosting SQLite Databases on GitHub Pages (2021) | Hacker News Reader

Hosting SQLite Databases on GitHub Pages (2021)

phiresky.github.io

86 pointsWA9ACE19 comments

It would be a it more compelling if every example, at least for me, didn't return `[error: NetworkError: A network error occurred.]`

I'm assuming your using firefox, if so it's this bug[1] that basically prevents range requests from working. Basically firefox says it will accept gziped data even though it's a range request and github pages dudifly sends back an unreadable slice of a gziped file.

1. https://bugzilla.mozilla.org/show_bug.cgi?id=1874840

It doesn't work for me on safari either.

works fine on safari desktop for me

huh. is this due to ambiguity in whether you want gzipped content vs gzipped transport (of arbitrary content), and/or which range the bytes are requesting? I can see both being useful, but I don't know what headers are available for these intentions...

my understanding is that technically only gzipped content is supported, not gzipped transport of arbitrary content. Due to ambiguity around the word 'append' [1] in the spec, firefox adds 'identity' (aka don't compress) to the end of the list of compressions supported while most other browsers replace the list with 'identity'. Also it should be noted that this is not a user configurable header so you can't actually try to override it.

There is a second layer to the bug in that github pages should almost certainly not be sending back slices of compressed files even if gzip is listed before identity and some change to something in the github stack probably exposed the bug that was there in firefox all along.

I literally just stumbled on this last week while doing a side project that involved browser range requests so this is fresh in my head.

1. it comes down to whether it was meant that 'Accept-Encoding:identity' should be appended to the list of header values possibly overwriting the one that already was there or if 'identity' should be appended to the list of values already in the 'Accept-Encoding' header. Firefox does the latter, everyone else does the former.

this exact same example used to work in firefox a few years back, i guess some change introduced this bug in between

Indeed, I lost the history in a shuffle, but a similar use case broke in some Firefox update, and it's the exact reason behind this comment:

https://github.com/seligman/podcast_to_text/blob/master/sear...

In my case, loading the entire file is loading a tiny bit more data, so this fallback doesn't hurt, but it's still annoying, and broke any hope I had of doing something more clever with the dataset.

The actual regression might be with githup pages where firefox was sending the same ambiguous headers the whole time but something in github's stack started interpreting them differently.

I get "Error connecting to database: RuntimeError: abort(Error: server uses gzip or doesn't have length). Build with -s ASSERTIONS=1 for more info" from the sposorship stats site lined toand the same as you for the demo.

its interesting, but it is not robust and I very much doubt it will have the claimed benefit for lasting without further work. At the very least its more work and requires cross browser testing.

It seems a lot of effort just to avoid paying a small amount a month for a VPS. Its free and " I forget about the backend and stop paying for whatever VPS it was on" stops being a problem (although running all the little things you have on one VPS would also simplify that).

I wonder if the author would use DuckDB WASM now?

is it doing HTTP range request, to not download an entire db?

I'm using DuckDB WASM on github pages. This will take about 10 seconds to load [1] and shows business trends in my county (Spokane County). This site is built using data-explorer [2] which uses many other open-source projects including malloy and malloy-explorer. One cool thing... if you use the UI to make a query on the data - you can share the URL with someone and they will see the same result / query (it's all embedded in the URL).

[1] - https://mrtimo.github.io/spokane-co-biz/#/model/businesses/e... [2] - https://github.com/aszenz/data-explorer

From "Show HN: TeaTime – distributed book library powered by SQLite, IPFS and GitHub" https://news.ycombinator.com/item?id=42264274 :

>> phiresky/sql.js-httpvfs: https://github.com/phiresky/sql.js-httpvfs

>> mmomtchev/sqlite-wasm-http: https://github.com/mmomtchev/sqlite-wasm-http

>> This project is inspired from @phiresky/sql.js-httpvfs but uses the new official SQLite WASM distribution

duckdb/duckdb-wasm: https://github.com/duckdb/duckdb-wasm

"PSA: SQLite WAL checksums fail silently and may lose data" so that's probably not how to sync sqlite; https://news.ycombinator.com/item?id=44672902

electric-sql/electric: https://github.com/electric-sql/electric :

> Specifically, Electric is a read-path sync engine for Postgres. It syncs data out of Postgres into ... anything you like. The core sync protocol is based on a low-level HTTP API. This integrates with CDNs for highly-scalable data delivery.

electric-sql/pglite: https://github.com/electric-sql/pglite :

> Embeddable Postgres with real-time, reactive bindings.

"Using Postgres for Everything" https://news.ycombinator.com/item?id=40420474

It's old news, but you can even host ClickHouse databases on GitHub pages: https://github.com/ClickHouse/web-tables-demo

Where does the 600mb SQLite file live though? GitHub won’t host that. Lfs server?

(2021)

>(or IPFS or any static file hoster)

Is it just me or It does not run on Firefox. I ran on edge it works. In Firefox I get: [error: NetworkError: A network error occurred.]

Reply on news.ycombinator.com