Building a highly-available web service without a database
blog.screenshotbot.io
blog.screenshotbot.io
The in-memory state can be whatever you want, which means you can build up your own application-specific indexing and querying functions. You could just use sqlite with :memory: for the Raft FSM, but if you can build/find an in-memory transaction store (we use our own go-memdb), then reading from the state is just function calls. Protecting yourself from stale reads or write skew is trivial; every object you write has a Raft index so you can write APIs like "query a follower for object foo and wait till it's at least at index 123". It sweeps away a lot of "magic" that normally you'd shove into a RDBMS or other external store.
That being said, I'd be hesitant to pick this kind of architecture for a new startup outside of the "infrastructure" space... you are effectively building your own database here though. You need to pick (or write) good primitives for things like your inter-node RPC, on-disk persistence, in-memory transactional state store, etc. Upgrades are especially challenging, because the new code can try to write entities to the Raft log that nodes still on the previous version don't understand (or worse, misunderstand because the way they're handled has changed!). There's no free lunch.
That's the basic design that rqlite[1] had for its first ~7 years. :-) But rqlite moved to on-disk SQLite, since with WAL mode, and with 'PRAGMA synchronous=OFF' [2], it is about as fast as writing to RAM. Or at least close enough, and I avoid all the limitations that come with :memory: SQLite databases (max size of 2GB being one). I should have just used on-disk mode from the start, but only now know better.
(I'm guessing you may know some of this because rqlite uses the same Raft library [3] as Nomad.)
As for the upgrade issue you mention, yes, it's real. Do you find it in the field much with Nomad? I've managed to introduce new Raft Entry types very infrequently during rqlite's 10-years of development, only once did someone hit it in the field with rqlite. Of course, one way to deal with it is to release a version of one's software first that understands the new types but doesn't ever write the new types. And once that version is fully deployed, upgrade to the version that actually writes new types too. I've never bothered to do this in practise however, and it requires discipline on the part of the end-users too.
[2] This might sound dangerous but in the current design of rqlite, the underlying SQLite database is completely rebuilt from the Raft log on startup (which is fsync'ed on every write). So any corruption of the SQLite database due power loss, etc is moot since the SQLite database is not the authoritative store of data in rqlite.
Yeah, I saw the recent post about reducing rqlite disk space usage. Using the on-disk sqlite as both the FSM and the Raft snapshot makes a lot of sense here. I'm curious whether you've had concerns about write amplification though? Because we have only the periodic Raft snapshots and the FSM is in-memory, during high write volumes we're only really hammering disk with the Raft logs.
> Do you find it in the field much with Nomad? I've managed to introduce new Raft Entry types very infrequently during rqlite's 10-years of development, only once did someone hit it in the field with rqlite.
My understanding is that rqlite Raft entries are mostly SQL statements (is that right?). Where Nomad is somewhat different (and probably closer to the OP) is that the Raft entries are application-level entries. For entries that are commands like "stop this job"[0] upgrades are simple.
The tricky entries are where the entry is "upsert this large deeply-nested object that I've serialized", like the Job or Node (where the workloads run). The typical bug here is you've added a field way down in the guts of one of these objects that's a pointer to a new struct. When old versions deserialize the message they ignore the new field and that's easy to reason about. But if the leader is still on an old version and the new code deserializes the old object (or your new code is just reading in the Raft snapshot on startup), you need to make sure you're not missing any nil pointer checks. Without sum types enforced at compile time (i.e. Option/Maybe), we have to catch all these via code review and a lot of tedious upgrade testing.
> it requires discipline on the part of the end-users too.
Oh for sure. Nomad runs into some commercial realities here around how much discipline we can demand from end-users. =)
[0] https://github.com/hashicorp/nomad/blob/v1.8.2/nomad/fsm.go#... [1] https://github.com/hashicorp/nomad/blob/v1.8.2/nomad/fsm.go#...
I mean, yes, the more disk IO rqlite has to make to more write performance will be affected. However the advantages of running with an on-disk SQLite database are worth it I believe. In addition rqlite supports storing the SQLite database file on a memory-backed filed system if users really want that[1]. That can help squeeze more write throughput out of rqlite.
>My understanding is that rqlite Raft entries are mostly SQL statements (is that right?).
That's right, rqlite does statement-based replication, though I'm currently looking into extending it so it also does changeset[2] replication where it makes sense.
[1] https://rqlite.io/docs/guides/performance/#use-a-memory-back...
Yes indeed! But this doesn't apply to a startup in the Explore phase, where you don't need replication, and how we did it for a long time. This is the phase where this architecture is the most use for product iteration.
But you're right, once you start using replication in the Expand phase, there certainly are engineering challenges, but they're all solvable challenges. It might help that in Common Lisp we can hot-reload code, which makes some migrations a lot easier.
That's no longer true, with modern desktop and mobile apps often using a database (usually SQLite) because relational data storage and queries turn out to be pretty useful in a wide range of applications.
Second, querying the RDBMS has been much simplified in past 20 years. We have all kind of ORMs and row mappers to reduce the boilerplate.
We also got advanced features like FTS which are useful for desktop and mobile apps.
Today it's a good choice to use RDBMS for desktop apps.
Is there an alternative? I haven't seen a "local filesystem is okay as data storage" software in the 21th century.
Well, there were "options" other than KV stores - MySQL launched a month before Viaweb (but flakey for a good long while.) Oracle was definitely around (but probably $$$$.) mSQL was being used on the web and reasonably popular by 1995 (cheap! cheerful! not terrible!)
(definitely understand making your own in-memory DB in 1995 though)
To be able to read and write you need native os apis, aka read() and write() otherwise how do you do it ?
that was all the rave for prototypes back then.
I think that is just written to disk as something like file41207393 when you click reply.
When the system needs an item it sees if it's cached in memeory and otherwise reads it from disk and I think that is pretty much the whole memory system. Some other stuff like user id that works in the same sort of way.
After reading the link, I don't think that database means the same thing for everyone.
The vwfaq still mentions loading data from disk, and also mention "start up a process to respond to an HTTP request." This suggests that by "database" they meant a separate server dedicated to persist data, and having to communicate with another server to fetch that data.
Obviously, this leaves SQLite out of this definition of database. Also, if you're loading data from disk already, either you're using a database or you're implementing your own ad-hoc persistence layer. Would you still consider you're using a database if you load data from SQLite at app start?
The problem with this sort of mental model is that it ignores the fact that the whole point of a database is to persist and fetch data in a way that is convenient to you without having to bother about low-level details. Storing data in a database does not mean running a postgres instance somewhere and fetching data over the web. If you store all your data in-memory and have a process that saves snapshots to disk using a log-structured data structure... Congratulations, you just developed your own database.
The web has gotten bigger and a lot of these practices simply would not fly today. If I was pushing a live fix on our prod machine with the amount of testing doing it live while on the customer is on the phone entails today, a good portion of you would be questioning my sanity.
Worse is better until you absolutely need to be less worse, then you'll know for sure. At that point you'll know your pain points and can address them more wisely than building more up front.
For server-based database engines you can still make an argument on shedding network calls. It's dubious, but you can.
What's baffling is that the blogger tries to justify not picking up SQLite claiming it might have features that they don't need, which is absurd and does not justify anything.
The blog post reads like a desperate attempt to start with a poor solution to a fictitions problem and proceed to come up with far-fetched arguments hoping to reject the obvious solution.
Wondering if my thinking is flawed, or if going this—arguably unnecessary—extra mile is part of the product and being successful in the space.
If your load fits entirely on one server, then just run the database on that damn server and forget about “special architectures to reduce round-trips to your database”. If your data fits entirely in RAM, then use a ramdisk for the database if you want, and replicate it to permanent storage with standard tools. Now that’s actually simple.
So, basically, https://litestream.io/ . But perhaps faster switching thanks to an explicit Raft setup? I'm not a litestream user so I'm not sure about the subtleties, but it sounds awfully similar.
That overly-simplified summary aside, I quite like the idea and I think the post does a pretty good job of selling the concept. For a lot of systems it'll scale more than well enough to handle most or all of your business even if you become abnormally successful, and the performance will be absurdly good compared to almost anything else.
- In GitHub readme you mention etcd / consul. Is rqlite suitable for transaction processing as well ?
- I am imagining a dirt simple load balancer over two web servers. They are a crud app backed onto a database. What is the disadvantages of putting rqlite on each server compared to say having a third backend database.
If you need to ensure that a given API request (which can contain multiple SQL statements) is atomically processed (all SQL statements succeed or none do) that is supported however [1]. That's why I think of rqlite as closer to the kind of use cases that etcd and Consul support, rather than something like Postgres -- though some people have replaced their use of Postgres with rqlite! [2]
[1] https://rqlite.io/docs/clustering/general-guidelines/#cluste...
[2] https://rqlite.io/docs/faq/#can-any-node-execute-a-write-req...
The main caveat here is that rqlite is an out-of-process database, which you communicate with over http. That puts it on similar grounds as e.g. postgres, just significantly lighter weight, and somewhat biased in favor of running it locally on every machine that needs the data.
So minimum read latency is likely much lower than postgres, but it's still noticeable when compared to in-process stuff, and you lose other benefits of in-process sqlite, like trivial extensibility.
I fail to see how this little bit of saving justifies all the complexity for run-of-the-mill web services that fit on one or a few servers as described in the article. The context isn't large scale services where 1ms/request saving translates to $$$, and the proposal doesn't (vertically) scale anyway.
The only thing in my comment that’s not directly based on the article is a handwavy 1ms/request saving estimate, and since they don’t provide any measurement, it’s anyone’s guess.
The guideline specifically advises to do what GP did: Instead of commenting whether or not someone read the article, to tell them that article answers their questions.
But if the blogger learned SQLite, how would they have a topic to blog about?
Also, no benchmarks. It's quite odd that an argument grounded on performance claims does not bother to put out any hard data comparing the output of this project. I'm talking about basic things like how does this contrived custom ad-hoc setup compare with vanilla, out-of-the-box SQLite deployment? Which one performs worse and by how much? How does the performance difference reflect in request times and infrastructure cost? Does it actually pay off to replace the dozen lines of code of on boarding SQLite with a custom, in-development, ad-hoc setup? I mean, I get the weekend personal project vibe of this blog post, but if this is supposed to be a production-minded project then step zero would have been a performance test on the default solution. Where is it?
I probably did a bad job then, because everything in the blog post was meant to be developer productivity claims, not performance claims. (I come from a developer productivity background. I'm decent at performance stuff, but it's not what excites me, since for most companies my size performance is not critical as long as it scales.)
The OP starts out by talking about periodically dumping everything in RAM to disk. I’d say that’s your checkpointing.
Just try and cold-start your database and run a fairly large select twice.
Even if you weren't using erlang, his influence (and in general, ericsson) permeates the telecom industry.
There are libraries available to wrap your stuff with this algorithm, and the benefit is that you write your server like it would run on a single machine, and then when launching it in prod across multiple, everything just works.
A transactional database is simple in Expand and Extract, but adds additional overhead during the Explore phase, because you're focusing on infrastructure issues rather than product. Data reliability isn't critical in the Explore phase either, because you just don't have customers, so you just don't have data.
Having everything in memory with bknr.datastore (without replication) is simple in the Explore phase, but once you get to Expand phase it adds operational overhead to make sure that data is consistent.
But by the time I've reached the Expand phase, I've already proven my product and I've already written a bunch of code. Rewriting it with a transactional database doesn't make sense, and it's easier to just add replication on top of it with Raft.
Unless, of course, your startup is in the business of selling DBMSes.
I think sql db make the most sense during the explore phase and you switch off of them once you know you need an improvement somewhere (like latency or horizontal scalability).
And this comes to the difference between Explore phase and Expand phase.
In the Explore phase, data migration was just running code on the production server via a REPL. Some migrations such as adding/removing fields are just a matter of hot-reloading code in CL, so there weren't a lot of migrations that we had to manually run.
In the Expand phase, once you add replication, this does become hard and we did roll out our own migration framework. But by this point we already have a lot of code built out, so we weren't going to replace it with a transactional database.
Essentially, we optimized for the Explore phase, and "dealt with the consequences" in the Expand phase (but the consequences aren't as bad as you might think).
Then you get used to near-zero latency that in-RAM data gives you, and when it outgrows your RAM, it's a pain in the butt to move it to disk :)
The premises are weak and the claims absurd. The author uses overstatement of the difficulties of serialization just to make their weak claim stronger.
These kind of people usually suck to work with. I'm glad they've found a startup to sink so I don't have to deal with them.
I usually start by putting those items into YAML files in a "data" directory. Actually a custom YAML dialect without the quirks of the original. Each value is a string. No magic type conversions. Creating a new item is just "vim crunches.yaml" and putting the data in. Editing, deleting etc all is just wonderfully easy with this data structure.
Then when the project grows, I usually create a DB schema and move the items into MariaDB or SQLite.
This time, I think I will move the items (exercises) into a JSON column of an SQLite DB. All attributes of an item will be stored in a single JSON field. And then write a little DB explorer which lets me edit JSON fields as YAML. So I keep the convenience of editing human readable data.
Writing the DB explorer should be rather straight forward. A bit of ncurses to browse through tables, select one, browse through rows, insert and delete rows. And for editing a field, it will fire up Vim. And if the field is a JSON field, it converts it to YAML before it sends it to Vim and back to JSON when the user quits Vim.
Now, with cheap hardware, they’re going back in time to the benefits of clustered, NUMA machines. They’ve improved on it along the way. I did enjoy the article.
Another trick from the past was eliminating TCP/IP stacks from within clusters to knock out their issues. Solutions like Active Messages were a thin layer on top of the hardware. There’s also designs for network routers that have strong consistency built into them. Quite a few things they could do.
If they get big, there’s hardware opportunities. On CPU side, SGI did two things. Their NUMA machines expanded the number of CPU’s and RAM for one system. They also allowed FPGA’s to plug directly into the memory bus to do custom accelerators. Finally, some CompSci papers modified processor ISA’s, networks on a chip, etc to remove or reduce bottlenecks in multithreading. Also, chips like OpenPiton increase core counts (eg 32) with open, customizable cores.
This exists in sufficiently mature Actor model[0] implementations, such as Akka Event Sourcing[1], which also addresses:
> But then comes the important part: how do you recover when your process crashes? It turns out that answer is easy, periodically just take a snapshot of everything in RAM.
Intrinsically and without having to create "a new architecture for web development". There are even open source efforts which explore the RAFT protocol using actors here[2] and here[3].
0 - https://en.wikipedia.org/wiki/History_of_the_Actor_model
1 - https://doc.akka.io/docs/akka/current/typed/persistence.html
If you choose say Cosmos DB, MongoDB or DynamoDB as your persistence provider you can even query the persisted state.
https://learn.microsoft.com/en-us/dotnet/orleans/grains/grai...
https://learn.microsoft.com/en-us/dotnet/orleans/grains/tran...
https://learn.microsoft.com/en-us/dotnet/orleans/grains/even...
But no, just more lispers.
I think this has to be the number one misunderstanding for developers.
Yes, SSD in terms of throughput or IOPs has gone up by 100 to 10000x. vCPU performance per dollar has gone up by 20 - 50x. We went from 45/32nm to now 5nm/3nm, and much higher IPC.
But RAM price hasn't gotten anywhere near the same fall as CPU or SSD. It may have gotten a lot faster, you may be even getting to stick lots of memory with higher density chip and channels went from dual to 8 or 12. But if you look at the DRAM Spot price since 2008 to 2022, you will see the lowest DRAM price has been the same at around $2.8/GB for three times. As the DRAM price goes in cycle with $8 / $6 per GB in between this same period. i.e Had you bought DRAM at its lowest point or its highest point during the past ~15 years your DRAM would have cost roughly the same plus or minus 10-20% ignoring inflation.
It was only until Mid 2022 it finally broke through the $2.8/GB barrier and collapse close to $1/GB before settling on ~ $2/GB for DDR5.
Yes you can now get 4TB RAM on a server. But it doesn't mean DRAM are super cheap. Developers on average or for those in big Tech are now earning way more than they were in 2010. Which makes them think RAM has gotten a lot more affordable. In reality even in the lowest point over past 15 years you only get at best slightly more than 2x reduction in DRAM price. And we will likely see DRAM price shot up again in a year or two.
A simplistic example, if a given node was limited to 16GB of RAM 20 years ago, I would need 256 nodes to have 4TB of RAM for my system (not including overhead for each OS).
Compared to today, where a single node can have that entire 4TB all in one chassis.
The total cost of RAM chips themselves may not have changed, but the actual cost of using that RAM in a physical system has dropped dramatically.
https://emoji.boats/ is the most public facing of these.
I also have built a whole class of micro-services that pull their entire dataset from an API on start up, hold it resident and update on occasion. These have been amazing for speeding up certain classes of lookup for us where we don't always need entirely up to date data.
I agree, the infrastructure required to make this happen eventually gets quite complicated. But the developer experience is what's super simple. If somebody had to take all our infrastructure and just use it to build their next big app, they can get the simplicity without worrying about the internal plumbing.
When I asked people working on it if they considered Redis or Mongo or Postgres with jsonb columns, they just said they considered all of those things but decided to roll out their own db anyway because "they understood it better".
This article gives off the same energy. I really hope it works out for you, but IMO spending innovation tokens to build a database is nuts.
AWS and other cloud providers are money printers because a lot of engineers are insanely tied into established patterns of doing things and can't think through things at a fundamental level. Ive seen company backends where their entire AWS stacks could be replaced by a 2 EC2 instances behind a load balancer with a domain name, without affecting business flow.
We did something similar to the work in the OP post at my work, we had a bunch of ECS tasks for a service, where the service did another call to an upstream service to fetch some intermediate results. We wanted to cache results for lower response latency. People were working to set up a Redis cluster. Except the TPS of the service was like 0.1.
Took me one day to code a /sync api endpoint, which was just a replica of the main endpoint. The only difference is that the main endpoint would spin of a thread to call the /sync endpoint, whereas the /sync endpoint didn't. Both endpoints ended with caching the results in memory before returning. Easy as day, no additional infra costs necessary.
But overall, personally, I don't hate the "spending innovation tokens to build a database is nuts" sentiment too much, because it keeps me employed at high salary while doing minimal work, where things that really should be basic CS are considered innovation.
Raft does consensus. Raft does not do persistence to disk, WAL, crash recovery, indexing, vacuuming (you're using tombstones for your deletes, right?), or any of the other necessary pieces of a database. That's not mentioning how such a system has no query engine, so every piece of data you're looking up in every place you need data is traversing your bespoke data structures.
What you described isn't a database. Keeping some disposable values cached isn't a database.
What you need from your side (and there are libraries that already do this): a) A mechanism to snapshot all the data b) An easy in-memory mechanism to create indexes on fields--not strictly needed, but definitely makes things a lot more easier to work with.
Bespoke data structures are just simple classes, so if you're familiar with traversing simple objects in the language of your choice, you're all set. You might be over-estimating the benefits of a query engine (and I have worked at multiple places that used MySQL extensively, and used MySQL to build heavily scaled software in the past).
It simply does not. The paper that definitionally is Raft doesn't tell you how to interact with durable storage. The raft protocol handles crash recovery in so far as it allows one or more nodes to rebuild state after a crash, but Raft doesn't talk about serialization or WAL or any of the other things you inevitably have to do for reliability and performance. It gives you a way to go from some existing state to the state of the leader (even if that means downloading the full state from scratch), but it doesn't give you a way to go from a pile of bits on a disk to that existing state.
If you have a library that implements Raft and gives you those things, that's not Raft giving you things. And that library could just be SQLite.
> You might be over-estimating the benefits of a query engine
No, I'm not. It's great to describe the data I want and get, say, an array of strings back without having to crawl some Btrees by hand.
That's being a bit pedantic. Yeah, I did mean that any respectable library implementing Raft would handle all of this correctly.
> without having to crawl some Btrees by hand.
This is not how I query an index. First, we don't even use Btrees, most of the times it's just hash-tables, and otherwise it's a simpler form of binary search trees. But in both cases, it's completely abstracted away in library I'm using. So if I'm trying to search for companies with a given name, in my code it looks like '(company-with-name "foobar")'. If I'm looking for users that belong to a specific company, it'll look like '(users-for-company company)'.
So I still think you're overestimating the benefits of a query engine.
The point of Raft is that you write your service like it was a single instance, using SQLLite or non relational equivalent, and then use Raft to run a distributed system that can have redundancy, all without additional infra involved, and for the vast majority of the use cases (i.e some backend or some web app service at a startup), this is more than enough, considering there is enough low level stuff in drivers and kernels to make data reliability pretty high already.
Ah, there it is! The "draw the rest of the fucking owl"
Magic!
I did this sort of thing recently. I felt bad doing it, I still objectively hate it, because I do know enough to know that basically I'm re-implementing what years of hardworking O/S developers have done, piecemeal. But at least I'm going in with my eyes open which feels better.
The only real mitigating factor I have is that the application is largely 'never-read' and then when reading is done, it's sequential batches. Which is not normally something databases optimise for and works okay for file-storage.
(If someone does know a lightweight database architecture that performs like this, let me know).
There are benchmarks out there proving that for some use cases (i.e. many small updates) where using SQLite is faster than using the filesystem. [1]
So not only do you get all of the benefits of a relational database, and literally centuries of engineering hours and bugfixes invested into SQLite, you might also get better performance (which is why I presume you even considered rolling your own).
The difference is that Redis costs extra in infrastructure.
The argument that im fighting agaist is that when someone says its more complex, what they mean is that they dont have experience in doing that. From a business perspective, this is something to consider when hiring from.an average pool, since you point about salary is correct, but the assumption that every single engineer fits this criteria is not correct.
Redis is/was basically just Tcl-typed which are persisted to disk using snapshots (Tcl commands) and append-only Tcl commands, that had a network protocol for non-Tcl applications to talk to
Reading posts like this makes me think the founders/CTO is mixing hobby programming with professional programming.
There's literally no reason to waste time doing all this.
So many lines of pointless, wasted code.
Which is absolutely fine if you are hobby programming but if you are running a business then this approach is wasteful.
https://youmightnotneedjquery.com/
The overwhelming majority underestimates the beauty and effort as well as experience that goes into abstractions. There are some true geniuses at times doing fantastic work, to deliver syntactical sugar while the critics mock the maybe somewhat larger bundle size for “a couple of lines frequently used.” That’s why.
In the end, a good framework is more than just an abstraction. It guarantees consistency and accessibility.
Try to understand the source code if possible before reinventing the wheel is my advice.
What maybe starts out to be fun quickly becomes a burden. If there weren’t any edge cases or different conditions, you wouldn’t need an abstraction. Been there, done that.
I haven't tried it out but just thinking of how many fewer organizational hoops I would have to jump through makes we want to try it out:
- No ordering a database from database operations.
- No ordering a port opening from network operations.
- No ordering of certificates.
- The above times 3 for development, test and production.
- Not having to run database containers during development.
I think the sweet spot for me would be in services that I don't expect to grow beyond a single node and there is an acceptance for a small amount of downtime during service windows.
With many enterprise databases these days, often "incremental" or other seemingly required backup modes are not included in the "community source" versions; perhaps because surely if you want your database to be backed up safely and then come back online safely, you certainly will fall into the "contact us for quote" enterprise customer demographic.
At least, with SQLite, copying even a hot (in-use) db file to a remote server will usually "just work", with the potential loss of a few transactions, but with most other database/servers, you definitely can't just backup the data directory occasionally and call it a day.
For stuff like MariaDB a quick search also finds options to perform snapshots, backups, restores etc.
And if you need to be super high available, set up a distributed DB like Cassandra - you lose the relational and transaction part, but at least you’re running a product with known failure modes and known ways to prevent/circumvent them.
I guess my bigger point is that besides “don’t roll your own crypto”, I’d also advice not to roll your own DB. There’s a lot of known stuff in the market, all built by people who made and fixed the mistakes you’re going to make a long time ago.
Sure you reduce deployment complexity, but what about maintaining your algorithm that implements data persistence and replication?
To assume that will never spectacularly bite you is naive. Tests also only go so far as you know what you are testing for, and while you don't know if your product will ever be used, you also don't know if it will explode in success and you will be hostage of your own decisions and technical debt.
These are HARD decisions. Hard decisions require solid solutions. You can surely try that with toy projects, but if I was in a position to build a software architecture for something that had a remote possibility of being used in production, I would oppose such designs adamantly.
1. I take it you've seen the LMAX talk [0], and were similarly inspired? :)
2. Are you familiar with the event sourcing approach? It's basically what you describe, except you don't flush to disk after editing every field, you batch your updates into a single "event". (you've come at it from the exact opposite end, but it looks like roughly the same thing).
The batching thing is something we can easily do with the library that I'm using. It allows us to define functions as arbitrary transactions. Within the transaction I can do anything that changes state, including changing multiple fields, so we don't have to keep flushing the log after every field.
1. Create your own in-memory database.
2. Make sure every transaction in this DB can be serialized and is simultaneously written to disk.
3. Use some orchestration platform to make all web servers aware of each other.
4. Synchronize transaction logs between your web servers (by implementing the Raft protocol) and update the in-memory DB.
5. Write some kind of conflict resolution algorithm, because there's no way to implement locking or enforce consistency/isolation in your DB.
6. Shard your web servers by tenant and write another load balancing layer to make sure that requests are getting to the server their data is on.
Simple indeed.
I don’t think we have anything to discuss here. He seems just to want to do cool stuff and his drop of databases seems to be because he just doesn’t know a lot of stuff there is to know.
I applaud attempt and might be that his needs will be covered by what he is doing.
But for everyone else yes, pick boring technology if you want to do startup because technology shouldn’t be hard or something you worry about if you are making web applications.
I'm not sure what qualifies as experience if Meta/Google doesn't. ;)
Just working at Meta/Google doesn’t impress me much just like Shania Twain would sing.
Just FYI, you are (perhaps unintentionally) showing your lack of experience.
There are many thousands of brilliant engineers for every brilliant engineer who also is a author/speaker/publisher. These are very different skills.
Also, perhaps the author _is_ the next Martin Fowler? You never know...
I am also not claiming I have a way to run every other new software compan.
According to LinkedIn:
- Masters in CS from UPenn
- 1 year as SWE at Google
- 6 years as SWE at FB/Meta
- 6 years running his own company
When I hear "not really experienced," I think recent college grad, not someone with a Master's and 15 years of industry experience.
Simple is what people are already using. And beware 'good for startups' tech. If you're successful you'll have legacy 'bad for scale' tech.
Surely this is a parody article of some sort?
We do use Preset for metrics and dashboards, and obviously Preset isn't going to talk to our in-memory database.
So we do have a separate MySQL database where we just push analytics data. (e.g. each time an event of interest happens.) We never read from this database, and the data is a schemaless JSON.
Preset then queries from this database for our metrics purposes.
But I have some bad news: you haven’t built a system without a database, you’ve just built your own database without transactions and weak durability properties.
> Hold on, what if you’ve made changes since the last snapshot? And this is the clever bit: you ensure that every time you change parts of RAM, we write a transaction to disk.
This is actually not an easy thing to do. If your shutdowns are always clean SIGSTOPs, yes, you can reliably flush writes to disk. But if you get a SIGKILL at the wrong time, or don’t handle an io error correctly, you’re probably going to lose data. (Postgres’ 20-year fsync issue was one of these: https://archive.fosdem.org/2019/schedule/event/postgresql_fs...)
The open secret in database land is that for all we talk about transactional guarantees and durability, the reality is that those properties only start to show up in the very, very, _very_ long tail of edge cases, many of which are easily remedied by some combination of humans getting paged and end users developing workarounds (eg double entry bookkeeping). This is why MySQL’s default isolation level can lose writes: there are usually enough safeguards in any given system that it doesn’t matter.
A lot of what you’re describing as “database issues” problem don’t sound to me like DB issues, so much as latency issues caused by not colocating your service with your DB. By hand-rolling a DB implementation using Raft, you’ve also colocated storage with your service.
> Screenshotbot runs on their CI, so we get API requests 100s of times for every single commit and Pull Request.
I’m sorry, but I don’t think this was as persuasive as you meant it to be. This is the type of workload that, to be snarky about, I could run off my phone[0]
Thanks for the comment! This is handled correctly by Raft/Braft. With Raft, before a transaction is considered committed it must be committed by a majority of nodes. So if the transaction log gets corrupted, it will restore and get the latest transaction logs from the other node.
> I’m sorry, but I don’t think this was as persuasive as you meant it to be.
I wasn't trying to be persuasive about this. :) I was trying to drive home the point that you don't need a massively distributed system to make a useful startup. I think some founders go the opposite direction and try to build something that scales to a billion users before they even get their first user.
I’m now completely lost as to why you believe this was a good idea over using something like MySQL/Postgres/Aurora. As I see it, you’ve added complexity in three different dimensions (novel DB API, novel infra/maintenance, and novel oncall/incident response) with minimal gain in availability and no gain in performance. What am I missing?
(FWIW, I worked on Bigtable/Megastore/Spanner/Firestore in a previous job. I’m pretty familiar with what goes into consensus, although it’s been a few years since I’ve had to debug Paxos.)
> I was trying to drive home the point that you don't need a massively distributed system to make a useful startup. I think some founders go the opposite direction and try to build something that scales to a billion users before they even get their first user.
This reads to me as exactly the opposite: overengineering for a problem that you don’t have.
For exactly the reasons you describe, I would argue the burden of proof is on you to demonstrate why Redis, MySQL, Postgres, SQLite, and other comparable options are insufficient for your use case.
To offer you an example: let’s say your Big Customer decides “hey, let’s split our repo into N micro repos!” and they now want you to create N copies of their instance so they can split things up. As implemented, you’ll now need to implement a ton of custom logic for the necessary data transforms. With Postgres, there’s a really good chance you could do all of that by manipulating the backups with a few lines of SQL.
Isn’t writing «a few Lines of SQL» also custom logic? The difference is just the language.
It is also possible that the custom data store is more easily manipulated with other languages than SQL.
SQL really is great for manipulating data, but not all relational databases are easy to work with.
Yeah. I hope it was clear in my post that the goal was developer productivity, not performance.
The round trip is only an issue on writes, and reads are super fast. At least in my app, this works out great. The writes also parallelize nicely with respect to the round trips, since the underlying Raft library just bundles multiple transactions together. Where it is a bottleneck is if you're writing multiple times sequentially on the same thread.
The solution there is you create a single named transaction that does the multiple writes. Then the only thing that needs to be replicated is that one transaction even though you might be writing multiple fields.
> it’s been a few years since I’ve had to debug Paxos
And this is why I wouldn't have recommended this with Paxos. Raft on the other hand is super easy for anyone to understand.
It claims to solve a bunch of problems by ignoring them. There are solid reasons why people distribute their applications across multiple machines. After reading this article I feel like we need to state a bunch of them.
Redundancy - what if one machine breaks either a hardware failure a software failure or a network failure (network partition where you can’t reach the machine or it can’t reach the internet)
Scaling- what if you can’t serve all of your customers from one machine ? Perhaps you have many customers and a small app or perhaps your app can use a lot of resources (maybe it loads gigs of data)
Deployment - what happens when we want to change the code and not go down if you are running multiple copies of your app you get this for cheap
There are tons of smaller benefits - right sizing your architecture What if the one machine you choose is not big enough you need to move to a new machine, with multiple machines you just increase the number of machines. You also get to use a variety of machine sizes and can choose ones that fit your needs so this flexibility allows you to choose cheaper machines
I feel like the authors don’t know why people invented the standard way of doing things.
Because we don’t want everything to fall over when one machine goes down we need at least 3 machines (for raft). So if our traditional db would have 500 GB of data we now need 3 machines with 500 GB of ram running at all times. That is an epic waste of money. Millions per year to run ? And you could store it in a db for a couple of dollars.
they store the index of files only in memory. and have the entire build time to fetch build-1 images to get ready for the diff.
it's much easier than most use cases
>
> And so, if your process crashes and restarts, it first reloads the snapshot, and replays the transaction logs to fully recover the state. (Notice that index changes don’t need to be part of the transaction log. For instance if there’s an index on field bar from Foo, then setBar should just update the index, which will get updated whether it’s read from a snapshot, or from a transaction.)
That’s a database. You even linked to the specific database you’re using [0], which describes itself as:
> […] in-memory database with transactions […]
Am I misunderstanding something?
I'm thrilled to see someone try something different, and grateful that he wrote about his positive experiences with it. Perhaps it will turn out to have been the wrong decision, but his writing about it is the only way we'll ever really know. It's so easy to be lulled into a sense of security by doing things the conventional way, and to miss opportunities offered by huge improvements in hardware, well-written open-source libraries, and powerful programming languages.
I have an especially hard time with the idea that SQL is where we all should end up. I've worked at Oracle, and I worked on Google AdWords when it was built on MySQL and InnoDB. I understand SQL's power, but I also understand how constraining it is, not only on data representation, but also on querying. I want to read more posts about people trying to build something without it. Redis is one way, but I'm eager to hear about others.
I wish the author good luck, and encourage him to write another post once Screenshotbot reaches the next stage.
I don't know if they're production grade. I was drawn to Braft because of Baidu's backing.
Sound similar to `stop the world Garbage collection` in Java. Does your entire processing comes to halt when you do this? How frequently do you need to take snapshots? Or do you have a way to do this without halting everything
But we aren't really taking the snapshot of RAM, instead we're running some code asking each object to snapshot itself into a stream. If you do this naively, it will block writes on the server until the snapshot is done (reads will continue to work).
But Raft has a protocol for asynchronous snapshots. So in the first step we take an immutable fast snapshot of the state we care about which happens quickly, then writes can keep going while in the background we serialize the state to disk.
Every single time… it’s always just the wheel being re-written.
2. Are network requests / other ephemeral things also saved to the snapshot?
Not sure what you mean by ephemeral things. If you mean things like file descriptors, they are not stored. Technically the snapshot is not a simple snapshot of RAM, it snapshots through all the objects in memory that are set up to be part of the datastore. (It's a bit more complicated and flexible than this, but that's the general idea.)
No transactions, no WAL, no relational schema to keep data design sane, no query planner doing all kinds of optimisations and memory layout things I don't have to think about?
You could say that transactions, for example, would be redundant if there is no external communication between app server and the database. But it is far from the only thing they're useful for. Transactions are a great way of fulfilling important invariants about the data, just like a good strict database schema. You rollback a transaction if an internal error throws. You make sure that transaction data changes get serialised to disk all at once. You remove a possibility that statements from two simultaneous transactions access the same data in a random order (at least if you pick a proper transaction isolation level, which you usually should).
> You also won’t need special architectures to reduce round-trips to your database. In particular, you won’t need any of that Async-IO business, because your threads are no longer IO bound. Retrieving data is just a matter of reading RAM. Suddenly debugging code has become a lot easier too.
Database is far from the only other server I have to communicate with when I'm working on user's HTTP request. As a web developer, I don't think I've worked on a single product in the last 4 years that didn't have some kind of server-server communication for integrations with other tools and social media sites.
> You don’t need crazy concurrency protocols, because most of your concurrency requirements can be satisfied with simple in-memory mutexes and condition variables.
Ah, mutexes. Something that programmers never shot themselves in a foot with. Also, deadlocks don't exist.
> Hold on, what if you’ve made changes since the last snapshot? And this is the clever bit: you ensure that every time you change parts of RAM, we write a transaction to disk. So if you have a line like foo.setBar(2), this will first write a transaction that says we’ve changed the bar field of foo to 2, and then actually set the field to 2. An operation like new Foo() writes a transaction to disk to say that a Foo object was created, and then returns the new object.
A disk write latency is added to every RAM write. It has no performance cost and nobody notices this.
I apologise if this comes off too snarky. Despite all of the above, I really like this idea — and already think of implementing it in a hobby project, just to see how well it really works. I'm still not sure if it's practical, but I love the creative thinking behind this, and a fact that it actually helped them build a business.
I like being able to draw a hard line between application data structures, often ephemeral and/or optimized for particular tasks -- and the persisted, domain data which has meaning beyond a specific application use case.
> Imagine all the wonderful things you could build if you never had to serialize data into SQL queries.
You can do all those "wonderful things" with an RDBMS too, it's just an additional step.
> First, you don’t need multiple front-end servers talking to a single DB, just get a bigger server with more RAM and more CPU if you need it.
You don't "need" that with a single DB too, you can also get a bigger machine. Also, you can use SQLite and Litestream.
> What about indices? Well, you can use in-memory indices, effectively just hash-tables to lookup objects. You don’t need clever indices like B-tree that are optimized for disk latency.
RDMBS provide all kind of indices. You don't need to build them in your code or re-invent clever solutions. They're all there. Optimized and battle-tested over decades.
> You also won’t need special architectures to reduce round-trips to your database.
You don't need "special architectures" at all. With the most simple setup you get thousands to requests per second and sub 5 ms latency. With SQLite even more. No need for async IO, threads scale well enough for most apps. Anyway, async is not a magical thing.
> You don’t need any services to run background jobs, because background jobs are just threads running in this large process.
How does this change when using an RDBMS?
> You don’t need crazy concurrency protocols, because most of your concurrency requirements can be satisfied with simple in-memory mutexes and condition variables.
I trust a proper proven implementation in SQLite or Postgres much more than "simple in-memory mutexes and condition variables". One reason why Rust is so popular is that it's an eye opener when the compiler shows you all your concurrency bugs you never thought you had in your code.
---------------------
RDBMS solve / support may important things the easy way
- normalized data modelling by refs and joins
- querying, filtering and aggregating data
- concurrency
- storage
Re-inventing those is most of the time much harder, error prone and expensive.
---------------------
The simplest, easy and proven way is still to use an RDBMS. Start with SQLite and Litestream if you don't want to manage Postgres, which is a substantial effort, I admit. Or cost factor, although something like Neon / Supabase / ... for small scale is much much much cheaper than the development costs for all the stuff above.
otherwise I get the messaging with edge you the database is the bottleneck
just need a one stop shop to do edge functions + edge db
https://www.microsoft.com/en-us/research/project/orleans-vir...