A non-trivial part of what doctors do is charting - where they strip out all the unimportant stuff you tell them unrelated to what they're currently trying to diagnose / treat, so that there's a clear and concise record.
You'd want to have a charting stage before you send the patient input to the LLM.
It's probably not important whether the patient is low income or high income or whether they live in the hood or the uppity part of town.