I have worked as an architect to the CTO of a 1200 person tech company for many years, and later as an architect over four teams at another tech company.
It is possible to learn good architecture, but I think when it's the most important is when it's the least sexy. For example, one of the most important things I learned at my last project was that one of the teams wasn't using database transactions for extremely important records. I learned this by noticing an oddity in the logs, then reading that area to prove it. Finally, I had to teach them how to use database transactions and what the different types meant.
While getting them to use transactions, I noticed they were POSTing a record to another team, then saving it in the database. What happened if the database INSERT failed? Extremely important data was now sent to another reporting system but not in the system of record. I helped them add a listener that would pull records from the database after it was inserted.
Now, what about the data in the other system, sure we fixed the source, but are there records there that were POSTed successfully, but not INSERTed? Time to find out by learning that system and checking. Sure enough there were 10ks of records that were orphaned, but showing up in reports to customers. These are financial records, it's extremely important that we not report on bad data. Cleaning this up was a six month project involving 25 people and three teams.
This was the kind of stuff that kills financial companies. The bad architecture would eventually have caused lawsuits and likely company collapse. However, most people weren't really that interested in it, the devs and product just wanted to get features out. I'd have to not only find the issues, but then be savvy enough to PowerPoint the issue to the right people to get the bandwidth to have these things fixed. It was thankless work, mostly tedious reading of code and logs and databases. Everyone helped but begrudgingly.
I will say the fastest way to learn to see these things is by learning to read code and document it in a way to help you think about it. I found the original issue about the POST and INSERT simply by reading the whole codebase two times top to bottom my first couple months on the job, documenting it all, and reviewing my documentation and thinking about it. Things like that just start to stand out as you ask "what happens if X fails" and follow it along to a logical conclusion. When I'd see something surprising, like weird logs, if seemed important I'd drill down until I found the cause.
Almost never in my experience as an architect have I been needed to design something up front. I actually think the job title should be changed to something like "cross team bug finder and fixer". So many times a team internally will be consistent enough, but once data crosses to another team the weird issues happen.
Another issue I found was that data was being stored in two databases. A check of a few lookup tables showed that someone had put the wrong status codes in one of the databases, so when a record was status 1 (OK) in one database, that meant something else in the other. Why were there two databases with the same data? Oh, a conversion that never finished, well let's finish it. Oh, no one still works here who remembers it, well I'll learn enough to figure out what is missing and finish the project, deprecating the old database.
I found another issue when I realized the devs on a team didn't realize splitting relational data between two databases was an issue. They couldn't have foreign keys, so they couldn't use joins in most cases. Because of this, the would fetch all records and join in memory. This was extremely inefficient, and they were certain that they needed a document database (and a two year rewrite) to deal with their performance issues. In reality, they only had a few 100ks of records, and I showed them that merging the two databases and using JOIN could fetch their reports and screens almost instantly.
Almost every dollar of value I've earned as an architect was like this. Nothing sexy, most of the time telling younger devs "no we don't need this new thing, it can be done with existing tools you already have in 1/10th the effort". Many times they want to rewrite everything, when in reality if they aren't good enough to see and fix these issues, they'll just recreate new ones.
This post makes me look like a database architect, and I think that's simply because many devs can do a decent job keeping code clean, but very few think about messaging, ACID, databases, scaling, and data. They can write unit tests and organize code into modules that are fine, but don't realize what should be cached and what shouldn't. They'll reach for solutions they just read about like microservices or horizontal scaling with messaging, but not even know how to get 1% of the benefit from their existing tech stack.
I really like the book Designing Data Intensive Applications for filling in the missing gap about data, transactions, and performance.
Here is my recommended reading list: http://deliberate-software.com/page/books/
I would encourage you to approach architecture like recommended here http://www.norvig.com/21-days.html. It will take a long time for the benefits to start, it's not something you can cram in a couple years. Learn to read code, document it, and read as much about code, data, and performance as you can.
Lastly, there's a lot of snake oil salesman out there who will teach you the "secrets" of architecture for a low fee of 10k at a one week conference they run alone. Udi Dahan, iDesign, etc. These guys will teach you one architecture, and will try to convince you a simple rewrite to this messaging first architecture will save everything. I've never seen any of those successfully applied, but I've seen devs go and become totally brainwashed until they can't even find obvious things in an existing system to improve, because they are so focused on trying to convince everyone to rewrite everything. I've seen great people be on a team, but never notice huge issues with the project, because all they can see is "it's not what Udi said it's good architecture".
Good architecture is 100% dependent on what the goal is, what the existing system is, who's working on it, and what the desired business outcomes are. A slow website isn't a big deal, if it's not hurting the business. Inconsistent data isn't a big deal in many different types of systems. It's all relative, so there's no One Perfect Design, and the only people saying otherwise are trying to sell you something.