How we built our startup (it's all about balance)
whimsical.co
whimsical.co
I hadn't heard of Terraform before, so thanks for the introduction. I have recently been attempting to familiarize myself better with systems infrastructure, and it looks interesting. I'll have to do some reading on it.
I have a question. You mentioned that Whimsical's back-end, "mostly serves as a lightweight layer between the client and the data store." Does this mean that you do most of the data munging client side so that once it arrives server side, most of the work is already done? And if so, what do the trade-offs look like in such a scenario? If I'm understanding correctly, this means the client is fatter, but there's less load on the server, meaning cheaper infrastructure costs. Is this a correct way of looking at it?
I'm just a junior dev trying to gain a better understanding of large systems, so forgive me for any ignorance or misunderstanding.
The backend serves as a fairly generic datastore with an API that allows data to be easily synced back and forth. It implements a logic (again non-specific to Whimsical) for how to handle concurrent updates (basically it implements CRDTs). I believe it's pretty similar to Google's Cloud Firestore - we might have chosen that if it was out of beta and we didn't mind the lock-in.
Two main reasons we did this: 1) to be able to easily add offline mode - we don't have it yet but it's fairly easy to add now 2) to minimize effect of network latency on user experience
As for less load on server & cheaper infrastructure costs - you are right, that is another benefit.
The complex part is how we sync actual data from clients to server while ensuring consistency. We basically store all data in a tree format and having a diffing logic how concurrent changes can be applied conflict-free in most cases. I'm considering writing a separate post on that.