That's not how it works. Whatever EB-5's don't get used eventually trickle down to EB-1, EB-2, and EB-3. They don't just go unused. If EB-6's start using them up, then there will be fewer EB-3's available every year.
if anything, prioritization is likely to work the other way: EB-1s first, then EB-2s, EB-3s, etc. see language below for EB-1 priority visa workers (with "extraordinary abilities"): http://en.wikipedia.org/wiki/EB-1_visa
note: i am not an immigration lawyer (IANA[I]L).
you may also wish to review language for the Immigration Act of 1990: http://en.wikipedia.org/wiki/Immigration_Act_of_1990
The question is what happens to those "extra" 9,000 visa numbers that are pre-reserved for EB-5. Everything I've been told says that whatever numbers aren't used by EB-5s gets "spilled over," starting at the top (EB-1), instead of going unused.
Also, I am not a lawyer either.