3 is probably too few. People don't really pay that much attention to what you wear, I promise.
I usually wear my shirts twice and take them to the dry-cleaner after. I'll only go once if it's been a hot day or if the shirt has dirt/stains. Maybe 3 times if I didn't wear it all day.
Plus I have about 7 pairs of high-quality pants that work well in an office and are comfortable for weekends. Most shirts go with most pants. So I have like 70 outfits so it really doesn't feel limiting, especially if you add in a few sweaters, jackets, and shoes to bring the combinations way up.
I did the math on this once to figure out cost-per-wear of my shirts. I kept rough track of how many times I wore a favorite shirt of mine before tossing it. Roughly 100 times. The shirt was like $120. I paid about $0.75 per wear in drycleaning. So like $1.75/wear.
(I could wash and iron for myself to bring this way down to like $1/wear but I hate ironing so I just pay the drycleaner - I also could have gone another 50 or so wears but I spilled coffee on it and I could always see the stain even though nobody else could I'm sure.)
Compare that with a cheap H&M shirt I bought a few years ago for like $40 - I've worn it twice and don't look forward to wearing it again since it doesn't fit as well and is a bit too trendy in its style for me. Cost is like $20/wear. Save your money and buy higher-quality clothes :)