It's a bit like perceived scarcity. Building a flashy portfolio website really only says two things: "I have free time on my hands", and "My demand is so low I have to put effort into advertising". Sure, if you're just getting started, that might be the brand you wanna build. But as you gain momentum, having less of a public brand is often better for business than having more.
It's counter-intuitive, I know; and maybe my experience was colored by the freelancers I happened to know at the time. But it convinced me enough that I took down my website while I was freelancing full time, I only put it back up after I quit, (to use as a resume to get full time job.) Removing my online presence did wonders for my business, and that's why I recommend it.