Why I Built Litestream
litestream.io
litestream.io
This is completely accurate. I've seen several teams do kubernetes, only to both spend 50% of their dev time on ops, AND cause outages due to kubernetes complexity. They do this all while boasting about zero downtime deployments. It's comical really.
From another thread on the home page right now:
The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance.
For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to apply cluster wide updates, one cluster at a time.
https://news.ycombinator.com/item?id=26106353
:-( :-(
Where did the KISS principle go?
A good weekend project and some learning ahead.
The tech aspires to provide that feature, and _technically_ can, but in practice you're always chasing the goose. (or spending so much on ops time/people that it becomes an invalid option, of course this becomes clear after you've fully invested in it)
Naturally it spreads like cancer. Non k8s native infrastructure is now abandon-ware. All that tech built over the last ten years is no longer seeing investment. Unfortunately, it solves real problems and rather well. Now a candidate to be reinvented and under the guise of reducing complexity it instead throws the users under the bus and actually does the opposite while costing a fortune.
When you step back and see: bare metals, vms, docker, k8s, stack of k8s plugins and tools, on prem and multi cloud all running concurrently... the IT team is really great at creating work and justifying their existence. Management needs to stop padding them on their back and hold them accountable for the mess they're generating.
It easy for me to complain, I guess, I'm not smart enough to understand how to kill this hydra. But I care about users and their experience and so maybe that's what's missing from this new frontier.
Once you said this, the is no longer a neccesity to say more. I could not agree more.
I just cp my sqlite file to S3 every 2 hours.
From my app, i have a page[1] where i can load any snapshot database saved on S3. I can backup at anytime too with a click, which i do before deployment.
P.s. Crave Cookie looks neat.
Talked about it on Indie Hackers podcast[2], where i briefly mention SQLite as well.
[2]: https://www.indiehackers.com/podcast/166-sam-eaton-of-crave-...
Always amazed by these niche businesses that make bank
we took down our revenue from indiehackers but somebody took a screenshot and posted on twitter of the stripe verified revenue. https://pbs.twimg.com/media/EXbNBVUX0AAotOo?format=jpg&name=...
"The software is built "from scratch" like the cookies and is part of Crave's success story. No other food company has the software Crave has for managing deliveries."
You should definitely do a write-up on Not Invented Here syndrom. And we sometimes "reinventing" the wheel, in moderation and for core components, really is the best solution.
So, $70k - $80k a month in profits
I'm too building a site on Crystal. This is my first production site(static comment). Would love to hear some story about how you run it.
Especially how you manage migration with Crystal+SQlite?
Funnily, been into F# the past few months and built a web framework for that too. Still in progress[2].
[1]: https://github.com/samueleaton/raze
[2]: https://wiz.run/
Congratulations my dude!
That is not normal native usage. If you make $50k, you're salaried at $50k, but you take home considerably less than that.
I guess in the end, it's just uncommon to say something like "Microsoft made $X billion last year" by itself, because it's just not clear what it means. Business news articles will almost always phrase something like that as "made $X billion in profits" etc.
Edit: It's also totally cool to ask for a clarification, e.g. "huh do you mean total annual revenue or this is your annual salary from your business?"
Modern machines are like a networked cluster themselves. You need to do a ton of work to tune both kernel and "hardware" parameters to identify bottlenecks with near-non-existing debugging tools.
There is one truth here: we used to get more performance by scaling "horizontally". Maybe it's time to "scale within"?
Do you have some benchmark results by any chance? Although it’s a bit of a can of worms, and I would understand if you didn’t want to get into it at this time.
This is why a database like SQLite has its miraculous read performance: "competing not with Postgres but with fopen()", as somebody said.
But any mutation locks the entire database. This, again, simplifies the implementation a lot, and guarantees serialized DML execution.
Rather few web apps need high write concurrency and low write latency. Great many web apps serve 99.9% of hits with SELECTs only, and when some new data needs to be persisted, the user can very well wait for a second or two, so rarely it happens. But this is often forgotten.
Same realization brought a wave of static site generators: updates are so rare that serving pages from a database makes no sense, and caching them makes little sense: just produce the "pre-cached" pages and serve them as is.
Maybe a similar wave can come to lighter-weight web apps: updates are so rare that you don't need Postgres to handle them. You can use SQLite, or Redis, or flat files as the source of your data, with massively less headache. Horizontal scaling becomes trivial. Updates are still possible, of course, you just pay a much lower complexity price for them, while paying a higher latency price.
It's easy to notice that horizontally scaled local databases are already known: it's called "sharding". Unless you need to run arbitrary analytical queries across shards, eventual replication of changes, where needed, is sufficient. (BTW this is how many very large distributed databases operate anyway.) Litestream already seems to support creation of read-only replicas. This can allow each shard have a copy of all the data of a cluster, while only being able to update its own partition. This is a very reasonable setup even for some rather high-load and data-packed sites.
Use WAL mode (writers don't block readers):
PRAGMA journal_mode = 'WAL'
PRAGMA temp_store = 2
PRAGMA synchronous = 1
PRAGMA cache_size = -64000
Not a great choice of API IMHO, but it’s a database, I’ve seen much worse.
conn.execute(f"PRAGMA cache_size = {-1 * 64_000}")
For us, the biggest thing would be that it can't be used if the database is on a network drive relative to the host process.
PRAGMA synchronous = NORMAL;
That avoids fsync() calls on the WAL until checkpointing. I don't know of any libraries that implement transaction coalescing as it can be different depending on if you need to batch sequential writes versus writes coming from multiple parallel threads
re: rqlite, yes it handles consensus/replication through Raft but IIRC it doesn't do sharding.
We build several products in Java using H2 as our database. What a pleasure. Full SQL support, zero configuration/installation/etc.
Just copy data-files/dbs around using normal file tools.
Just start up one process and your app is ready.
The main use I've put it to over the years is as an in-memory database for integration tests.
For a long time, it was possible/practical to use SQLite from Java. Now, it is, but not if you want to keep things as pure Java (and another commenter mentioned). But really, in my mind, that’s the only real benefit for H2, the fact that’s it’s pure Java. So if you need that, you’re good.
But otherwise, I try to stick to SQLite.
We've been doing it for 5 years now. Basic tricks we employ are:
Use PRAGMA user_version for purposes of managing automatic migrations, a. la. Entity Framework. This means you can actually do one better than Microsoft's approach, because you don't need a special unicorn table to store migration info. A simple integer compared with your latest integer and executing SQL in the range is all it takes.
Use PRAGMA synchronous=NORMAL alongside PRAGMA journal_mode=WAL for maximum throughput while supporting most reasonable IT recovery concerns. If you are running your SQLite application on a VM somewhere and have RTO which is satisfied by periodic hot snapshots (which WAL is quite friendly to), this is a more than ideal way to manage recovery of all business data while also giving good throughput to writers. If you are more paranoid than we are, then you can do FULL synchronous for a moderate performance penalty. This would be more for situations where your RTO requires the exact state of the system be recoverable the moment it lost power. We can afford to lose the last few minutes of work without anyone getting yelled at. Some modern virtualization technologies do help a lot in this regard. Running bare metal you need to be a little more careful.
For development & troubleshooting, being able to copy a .db file (even while its in use) is tremendously powerful. I can easily patch up a QA database I mangled with a bad migrator in 5 minutes by stopping the service, pulling the .db local, editing, and pushing it back up. We can also ask our customers to zip up their entire db folder so we can troubleshoot the entire system state.
Being able to use SQLite as our exclusive data store also meant that our software delivery process could be trivialized. We use zero external hosts, even localhost, for our application to be installed or started. We don't even require a runtime exist on the base operating system. Unzip our latest release build to a blank Win2019 server, sc.exe the binary path, net start the service, and it just works. Anyone can deploy our software because it's literally that simple. We didn't even bother to write a script because its a bigger pain in the ass to set powershell execution mode.
So, its not just about the core data storage, but also about the higher-order implications of choosing to use a database solution that can be wholly embedded within your application. Because of decisions like these, we don't have to screw around with things like Docker or Kubernetes.
https://levlaz.org/sqlite-db-migrations-with-pragma-user_ver...
I'm used to leaning on SQLite for desktop apps, but now I'm keen to think about it WRT to web apps using these tips.
I yet wait to see how somebody serve 1M users on a web service using sqlite. Sounds like you can do all that since you create desktop app, you almost never need anything more then sqlite for that.
How does virtualization help with data loss? I would expect that a VM can't have guarantees better than the underlying physical hardware provides.
E.g. storage virtualisation.
Can you say more about how (and which) modern virtualization technologies help? RTO is something I've never found a happy to, since piecing together any missing data is painful, but avoiding a clustered DB setup (or the cost of Aurora) is always welcome.
In my experience, sqlite performance becomes problematic quite quickly, even with settings you mentioned (WAL etc).
I should amend my original post, because I know a lot of developers fall into the trap of thinking that you should always do the open/close connection pattern with these databases. That is a huge trap with SQLite. If you want to add some extra zeroes to your benchmark figures, only use a single connection for accessing SQLite databases. Use application-level locking primitives, rather than relying on the database for purposes of getting consistent output from things like LastInsertRowId and in cases where transactional scopes are otherwise required. This alone can take you from 100 inserts/second to 10k without changing anything else.
You mean for generating unique primary keys? Why would last insert row id be slow?
> and in cases where transactional scopes are required
Could you elaborate on what you mean by this?
Transactional scopes meaning scenarios like debiting one account and crediting another. This is something you can also manage with locking with application-level primitives.
I’d assume the DB would be most efficient at handling it’s own. At least to the extend that it wouldn’t garner a 100x speedup to do it in app.
We need to remember that opening a connection to SQLite is like opening a file on disk. Creating/destroying file handles requires far more resources and ceremony than taking out a mutex on a file that is never closed.
Here is my SQLite compilation flags: https://github.com/liuliu/dflat/blob/unstable/external/sqlit...
Here is where the flag used: https://github.com/sqlite/sqlite/blob/d46beb06aab941bf165a9d...
My bet is on a single document server, and I'm writing a programming language for board games : http://www.adama-lang.org/docs/what-the-living-document
While I do see a future of the concepts I espouse as I'm making the language feel like an excel-reactive environment, my focus is on having fun. This year looks to be a slow year for the language as I built up some UIs for games.
Litestreem looks very interesting because it may be the missing piece where I could leverage it to get durability of a single database.
Let me know if you end up giving Litestream a try and how it works for you. I'm really working to make the developer experience as easy as possible and I'd love to hear feedback.
That said, I've always been a little worried about trying SQLite since I'm so used to Postgres. I've currently got Postgres running alongside my app in a Docker container, which isn't too hard to manage. I'm curious whether anyone has switched from Postgres to SQLite in a web app context (whether in the same project, or when making a new project) and if they've found themselves missing any of the features Postgres offers. I've tried to research this before but always found just googling "sqlite vs postgres" just results in surface-level differences that mostly focus on performance, whereas I'm more curious about e.g. the differences in their JSON extensions.
Regarding Postgres vs SQLite, I've found that I can use much simpler SQL calls with embedded databases when I don't need to worry about N+1 query performance issues. That makes many of the query features moot. That being said, there is a JSON extension for SQLite[2] although I haven't tried it.
I'm using multiples DBs and one of them is a simulation of key-value store. It's like a (Python) dictionary that is really an SQLite database, then I use keys like:
db["users:1000:email"] = "email@email.com"
Many people also mention concurrency, but I think that if you make your INSERT/UPDATE/DELETE statements fast and short + use WAL mode + use PRAGMA synchronous = 1, and some other optimizations, you can get quite far.
Since the web app runs on a single, always-on dyno, seems like it may work to use Litestream to (1) continuously replicate and (2) restore when we restart the dyno.
For folks who have dug into Litestream further than me, any thoughts on this use case?
(Of course, need to handle starting and supervising Litestream and the web app from one process, per Heroku's 1:1 process-dyno model.)
Coincidentally, I stumbled across an old tweet by Ben (OP/Litestream author) about Heroku and SQLite[1] a month ago, when first thinking of getting the app off Postgres.
[1]: https://twitter.com/benbjohnson/status/1186666174467039233
Edit: typos
Without advocating for Kubernetes, there's a pretty big difference between running a single application like WordPress or whathaveyou that receives only occasional updates and a SaaS application that is actively developed by hundreds or thousands of engineers deploying dozens of times per day. Yes, Kubernetes is complex and that complexity can introduce its own downtime issues, but that risk is a large constant whereas without it the risk increases with the number of deployments (and deploying larger deltas less frequently carries its own penalties). It's important to understand and acknowledge these dynamics in order to optimize for uptime and velocity.
- Disks are replicated to another zone on every write.
- Incremental disk snapshots can run once per hour and are stored in Cloud Storage.
This means when the OS fsync's it is actually copying those bytes to another zone.
Best of luck.
Does that mean I can use an in memory SQLite DB and have it replicated too? That would have some amazing potentials.
It's so simple, just don't have a lot of data that doesn't need to interact with other data and your problem is solved!
/s
Sarcasm noted though. :)
Actual concerns about running a real application with real uptime requirements.
1. Say your EC2 or docker container that's hosting this goes down. Is that left up to the user to deal with? RDS handles this for you
2. No ACID transactions if you ever outgrow a DB. You talk about in your pitch that the vertical scaling, so you have to just keep bumping the VPS/container memory.
3. Sure a SaaS application where a customer specific DB is isolated, but as soon as you hit any _real_ scaling limits you immediately are back to the entire problem statement you are aiming to (at least you hint at that in your pitch) solve which is the crazy n-tier architectures we have.
While I was being sarcastic, I was not being intellectually dishonest. There are entire hosts of problems that you call out you are trying to solve without providing any real solution.
Thanks! I appreciate it.
> Say your EC2 or docker container that's hosting this goes down. Is that left up to the user to deal with? RDS handles this for you
Yes, that's out of scope for Litestream since there are a lot of ways to manage that depending on your application. I agree that RDS wins here for simplicity.
> No ACID transactions if you ever outgrow a DB. You talk about in your pitch that the vertical scaling, so you have to just keep bumping the VPS/container memory.
You still have ACID transactions but they are just per-shard. For a SaaS application, that seems reasonable since they're localized to the customer (assuming you're sharding by customer).
> Sure a SaaS application where a customer specific DB is isolated, but as soon as you hit any _real_ scaling limits you immediately are back to the entire problem statement you are aiming to (at least you hint at that in your pitch) solve which is the crazy n-tier architectures we have. [...] There are entire hosts of problems that you call out you are trying to solve without providing any real solution.
I'm not trying to solve an infinite scaling problem. If you're seeing sustained 100K request/sec on your application then you'll need specific solutions. But I'd argue that 98% of applications never come near that threshold and those are the applications that could benefit from simpler architecture.
Thanks for all the feedback. I hope I'm not coming off as argumentative.
To be fair, these features of k8s are nice to have. Is there a similar tool for running single node (i.e. ec2) instances that offers zero-downtime deploys? One way you could do it is run a EKS cluster with a single ec2 node running your single-node setup. You could get the benefits of k8s while still running the simple single-node arch
systemd can be sufficient for a zero-downtime deployment.
[0] https://vincent.bernat.ch/en/blog/2018-systemd-golang-socket...
[1]: https://gohugo.io/
[2]: https://getdoks.org/
Why not advocate instead for people to start out running their application and postgresql on the same server, with wal-e for backups to S3?
That has almost the same benefits while everything fits on one server, but opens up alternative approaches to scaling if one day things no longer fit.
Yep. But don't tell anyone. This is a secret weapon/super power most juniors (and many mid/seniors) have been conditioned by the GOOG/FB/AMZN/MSFT approved project literature to believe is simply untrue. The number of experienced, highly skilled founder devs I know that reach for shiny tech stacks that solve issues they don't have but can "scale" is a 100:1. That choice comes with staggering costs.
I suppose single node+db works fine for hobby SAAS and expert programmers who can squeeze every bit of performance from the machine. But for the average in-efficient programmer or teams, standard PostGres as DB and HA services fronted by a load-balancer work better.
Your project is far more likely to die within a couple of years from having too few users than from having too many. If you find you're having scaling issues then that's a good time to solve them - your project is popular enough to justify the investment.
You'll likely be recoding your project quite a lot in the early phase of the development anyway as you learn more about the problem and (most importantly) more about what your users want/need. Anything that can reduce the cost of this early iteration will pay dividends later because it makes it more likely that you'll need to scale - and by the time you do need to scale you'll be scaling something that you understand and other people want.
Having dealt with complicated db replication this sounds like a good fresh idea.
[1]: https://tersesystems.com/blog/2020/11/26/queryable-logging-w...
See https://sqlite.org/src/doc/trunk/README.md and https://www.fossil-scm.org
I'm currently working on implementing a set of data structures (and more things later) on top of SQLite (https://github.com/litements/) for some of the same reasons you mention in the article.
I probably don't even have 1/10 of your experience, but your work is really motivating me to keep working on it, thanks!
One drawback I can think of is it would be difficult to create reports, cross customer.
Reports are nice because they can generally be done in an async job that's effectively a big map-reduce run across your DBs. If you need faster real-time reports you're going to want some kind of pipeline for events and stream processing that's outside the scope of this anyways.
If bedrockdb ( https://bedrockdb.com/ ) replicated to object storage like litestream, I’d be in heaven.
1. Applications which are read-heavy & serve fewer than 10,000 requests per second.
2. SaaS applications which can be sharded so that the largest customer uses 10,000 requests per second or less.
Once read-only replicas functionality is added, I'm also excited about globally-distributed applications where local servers can serve users with low-latency read requests. For example, you could run a high-traffic e-commerce site with 20 PoPs spread across the world for only $100/month. Customers in India could get the same fast response times as someone in the US. I think that's pretty compelling.
Is there a way to backup encrtypted? This would be a killier feature!
Is making the project closed to contributing part of that plan?
How did you decide on GPLv3 liscense?
Basically the argument is: do it yourself, and time spent learning how to do it effectively is compensated by the time not spent fighting over complexity caused by operating (and understanding how to use properly) stuff that allegedly "does it for you".
The argument sounds plausible in principle, but I'm not entirely sure if this argument resist the harsh clash with reality (where you end up with both doing it yourself and dealing with the effects of your own complexity but this time with no community to consult)
The main point here is that instead of reaching for a costly to maintain, to configure, to secure, etc. DBMS first, many folks would likely be better served by the vastly simplified management of a single SQLite process.
A lot of solutions don’t take into consideration “day 2” operations: mostly around no downtime upgrades.
No, SQLite has WAL and can recover from failures. The biggest problem with SQLite is concurrency. You can't write from multiple processes in parallel.
Litestream runs continuously on a test server with generated load and streams backups to S3. It uses physical replication so it'll actually restore the data from S3 periodically and compare the checksum byte-for-byte with the current database.
I made sure Litestream would be safe with the primary database so it only communicates through the SQLite API for locking and state. It should be completely safe to run unless there is a bug in SQLite itself. You can also continue to use a separate periodic backup strategy if you're not comfortable running Litestream alone for disaster recovery.
Why Amazon S3? Was it a huge invention by people at Amazon, and even if so, does it assign credit where it is deserved? Is Amazon helping the open web? I don't think so.
Because it's extremely cheap, durable and easy to read/write?
Even software such as Ceph calls themselves compatible with "Amazon S3 API".
Unfortunately Amazon has not done a very good job crediting predecessors, but they do deserve credit for bringing what had previously been rather niche ideas (I know because I was there throughout) to the masses.
It's bad from Amazon to not credit predecessors. If previous APIs are that close with S3, adding their branding is problematic.