You conflate different meanings of "context."

The problem Brooks describes around communication has to do with multiplying the number of paths as the team grows. The "surgical team" approach includes a member in charge of documentation at all stages, to give everyone access to the same context.

A small team may "vibe code" something they call a CRM in a week, but unlikely it will work and stand up to requirements changes. Only someone who doesn't understand the business domain would claim they can coax a solution out of an LLM in a few days.