Ask HN: How to get practical experience in developing scalable apps
My current job involve working in decade old product, how can I get a practical experience in developing a highly scalable and distributed applications.
2. Take something you've built, and imagine that it faces a specific scaling problem. For example, let's say the most complex thing you've built is a to-do list app. Maybe you can imagine some things that could make it grind to a halt when running as a monolith on a single server. e.g. what if a single user has 10,000 or 100,000 TODOs? Fake that situation (just insert many many fake TODOs into the database). Now see if you've created a scaling problem. If not, try a larger number, or a different dimension (e.g. what if there are many many simultaneous requests?) until you create a problem.
3. Look in that book again and pick a couple of ways you could solve the problem (database sharding across multiple servers? caching? pagination in your API calls?) and implement one of them.
(If you have difficult at step 2, you could try running your app on a virtual machine with very little RAM and disk, but imagine that's the largest machine type available.)
The problem with focusing on "scalable" is that wallets and budgets don't scale infinitely and hosting isn't free.
Being able to build stuff that is "scalable" is not enough. You need to be efficient which generally involves "scaling down", ie how much of this data can we be drop because it's useless and no one cares about it?
- If you wanna change configuration on the fly, in parallel, your app should support something like Prometheus reload: SIGHUP or http://<ip>/-/reload - rolling updates might be time consuming (you need to separate between boot config and dynamic config)
- Add feature flags to your app.
- Use proper secrets management from day one. Something like Vault or Credstash could go a long way.
- Perform migrations to the CD system, don't use init containers
... these are the ones I can think off the top of my head now, but there are more.
Most of the scalability stuff is about failure modes and how your system responds to it; so you don’t need stress tests for that.
Know the properties of your system. Eg what carries load? How does it scale? How much can it bare? Then mess with it on your cluster. Etc.
I suggest you start with Kubernetes and try and put together something like a self scaling PostgreSQL Citus cluster . Put data in it and then smash at it with big queries when ready.
Examples of such open source applications in use for distributed, large scale systems all over the world are tensorflow, elasticsearch, and hystrix; this is just the tip of the iceberg.
Contributions to OSS will also be beneficial if you need to find a job in the business.