I think another part is simply nomenclature. There's a joke about the only two hard problems in computer science: Cache expiry and naming things. It's a joke because cache expiry reduces to naming the keys correctly if you do it right.

Basically, if the different parts of the system don't have sensible names that represent what they actually do (and that there are, indeed, different parts of the system), then there's an architecture problem.

You have to have an effective metaphor for the way that you want the system to work, and the problems you want each part of it to solve, or nobody will be able to understand the entire thing.

I run into this often. People have a functional but struggling system designed with the "big ball of mud" pattern, where you just keep slapping gobs of code onto it until it does what you need, and then they want someone to come in and "do architecture" at it. This predictably fails.

So, there's your heuristic: if it has good names, and the pieces do what they say on the tin, you're on the right path. When something starts to have responsibilities that aren't under that name, you should move those responsibilities somewhere else and name them appropriately.

Edit: There's also a second-order level where you start to recognize the way that cross-cutting concerns happen in different areas and design a piece of the system to handle that concern in an organized way. You can probably think of a bunch yourself, but an easy example you want to handle earlier rather than later is logging, and another significant one is authentication

Part of it is indeed experience, but another part of it is just being able to visualize the entire application (domain experience helps with this) and see it as a series of interconnected modules. Think of the parts individually and what downsides they have and than as a whole.

An example of this is if you have a module that does something very GPU heavy or very bandwidth heavy. On an architecture layer you want want to give it it's own server outside of the main server (in the case of a web app).

In terms of code you need to think about how some modules will need to be updated in the future (want to support multiple payment methods) - and also to think what modules need to be secure or performant. If two modules will always exist together they might be something you can couple if it saves you a large amount of developer time. But if they might be re-used in the future they need their own API's and no close coupling.

There aren't many shortcuts one can take in that area -- the big difference between junior and senior developers is that the senior developers have experienced what can go wrong, often learning from their own mistakes and the mistakes of their employers, so they are better able to make design/architecture decisions.

If you're a junior developer and you don't have good senior people to learn from in your job, it's time to start looking for another job.

Unfortunately, most of the above knowledge isn't taught in schools and comes mostly from experience.

Seriously, though, come up with a hypothesis about how your system will scale, then devise an experiment to test the hypothesis. If the hypothesis fails, figure out why, fix it and repeat until you get the behavior you want.

The next important piece is monitoring and measuring the performance metrics important to you, then figure out how to change your architecture when the system grows enough to have new bottlenecks.

Having a good load test tool, for example, can be invaluable in learning how your system actually performs.

Some skill in general optimization-thinking is key. The ability to visualize how the entire system works is key. Aptitude for API and general architectural design is key.

And even when you get it all right, _something will fail._ At which point profiling and having the right metrics in place so you can see what's failing is key.

Also have a clear sense of direction of where you want to be going and what you want to be doing. Without that management and mentors won't know how to push you along and might come to expect you to remain in the same position for longer than you expect to.