People have constitutional rights, objects do not
The building was part of the person's effects. I don't understand how your argument is legal (not that I don't believe you - civil forfeiture has been going on long enough that I assume SCOTUS has heard a sampling of cases).
It beggars belief that this practice has stood for decades.
Another angle of defense might be to ask if it's even possible to violate the 6th amendment. What would violating the 6th amendment look like? I suspect the answer will closely resemble civil forfeitures.
If I use a payphone to order a hit, is it reasonable for the government to come rip it out of the ground? What if I use a freeway to smuggle drugs? Now the road must be torn out?
Money is fungible and has all sorts of uses. Any particular pile of cash is no more or less likely to be intrinsically criminal in nature. Just like phones and freeways.
I'm pretty sure that's been modified to US people on US soil, at least 100 miles from a border, have constitutional rights, but those rights were all intended to mean something different than what they actually say.