Or just use mature solution like Rails for example it solves most of typical problems and has easy itegrations. If you need more flexibility then maybe Node.js or other ecosystem is better.

Generally theme about architecture is called System Design and there are resources to learn it for interviews/learning purposes but most of it only is applicable at very big scale and you might never need it. Read "Designing Data Intensive Applications" and look into [1] if still interested.

IMO better to invest time into data design/querying (caching) so that you won't think you need some vodoo to handle your traffic. Like 90%+ of sites/apps don't have 1000req/second and this should be handled by Python on Arduino so as long as your DB design is good you are fine.

For async processing it's worth taking a look into Temporal [2] as it looks as promising alternative to job ques and cronjobs.

I can also recommend [3] to learn some cloud patterns

[1] https://github.com/donnemartin/system-design-primer

[2] https://temporal.io/

[3] https://docs.microsoft.com/en-us/azure/architecture/patterns...

There's usually no right or wrong answer - it's all about tradeoffs. I personally prefer to keep the stack and architecture as simple as possible, but many others prefer to rely on 3rd party code as much as possible, regardless of added complexity. Sometimes my choices bite me (I have to reinvent more wheels than I was expecting), sometimes the too-many-layers-of-abstraction approach will bite too (like errors you have no idea why they happen).

Compare different solutions available to the same problem, understand their tradeoffs, and then pick what better aligns to the needs of your project. If you're working with a team, you probably will not be making that decision alone, but the process could be the same.