So, documentation is the solution? But do people have time to document everything?
Documentation has to be done once. (Well, it has to keep being updated. But for any change, the documentation only needs updated once.) But new people need to know about it over and over, because as you grow, you keep getting new people. You don't have time for each new person to have to go exploring to find the answer to each question. It's far less time to have one experienced person write down the answer... if you have each new person read all the relevant documents, and if you have people update the documents every time anything changes.
The last point is important. It has to become part of the culture. It should also become part of the code review/commit process.
At my last job, I was on the wrong end of this. Much of the initial code was written by two people. One was now a director; the other was incredibly busy. The code was extremely object oriented, to the point that it was very hard to figure out where anything happened and therefore where changes should be made. The documentation was a number of UML diagrams.
The result was that new people (including yours truly) wasted huge amounts of time trying to find their way around in the code. Even after being there for three years (and with over 35 years of experience), I still found the code very hard to work with.
What that code needed was someone (one particular someone) to take a solid month and work on writing good beginner-to-intermediate documentation for the code base. It would have saved literally man-years of time for new people.
Any given day someone can win the lottery, inherit a lot of money, or need to leave for some other reason.
Every complaint they had was a direct effect of failure to document. Well that or the other people are just complete idiots ... you decide.
Giving people additional documented responsibilities helps. Lucy is responsible for API docs and swagger up to date and for building that out as business needs. Varun is responsible for coding practices docs. And so on. If something is out of date they update it. Use Confluence or Notion or something else good.
And to read everything. And to find it.
I find documentation is often meaningless or worse -- in networks you find people documenting IP addresses on static pages like confluence pages or powerpoint presentations. That isn't important. Sure it should be in a single source of truth -- emphasis on the single -- but if isn't, it's not the end of the world.
A new employee can see that router 1 is connected to router 2 by looking at the config. What they can't do is tell you why its connected. And why the route filters are the way they are. The how is easy to reverse engineer, but documentation should start with the why. Anyone changing something can then make an informed decision about whether the new solution still meets the why, or if the why isn't relevant any more.
Why are we using this architecture, why are we using an ec2 vs lambdas (or vice versa), why are we putting this infront of cloudfront and this direct, why is this in us-east-2 and this in eu-west-1.
A quick look at the teraform, or even manual things like "ec2 describe instances" tells you what's deployed, what the IP addresses are, what vpcs are connected.
It doesn't tell you why, that's where documentation comes in.
Business. If the customer can’t feel it it doesn’t exist.
Ops. If it isn’t automated it doesn’t exist.
Product. If it isn’t documented it doesn’t exist.
Engineering. If it isn’t in code it doesn’t exist.
You have an opinion about security? Have you written a linter? Is it documented? Is it in the deployment pipeline? Do customers care?
If it’s only in your brain I don’t accept it and refer you back to the existence rules.