Prime has become a bit less useful over the past year due to the introduction of Add-On Items.
One of the great utilities of Prime (and Amazon in general) is that it saves me a trip to Wal-Mart for smaller items. Need a pack of Sharpies? Ship it from Amazon, because a trip to Wal-Mart is an hour of my time wasted and a level of frustration that I really could do without. I'm willing to pay money for that convenience.
Amazon also sometimes has a "bulk" item at a higher price.
And to your point about attaching them to a prime order, I would have to imagine Amazon is aware of this limitation and I wouldn't be surprised if they add this feature soon.
Therefore Add-On items are just an improvement to Prime.
Also, I'm pretty sure everything that's actually sitting in an Amazon warehouse is prime-eligible. The stuff that isn't is the stuff going through 3rd party sellers that don't use fulfillment by Amazon.
Everything in the warehouse is not prime. If fact, an item can flip from prime to not prime in near real time if the most accessible inventory is used up and all that is left is the stuff in the 'reserve' areas of the warehouse. This is the area where the giant boxes of inventory stored. Grabbing inventory from there involves pulling it down, opening, pulling apart eaches, and then stowing in fast access areas.