There's a series of cases going way back. The first one had to do with the seizure of a vessel used by a Spanish Privateer (The Palmyra, 25 U.S. 12 Wheat. 1 1 (1827)). Although not particularity well reasoned -- it doesn't even analyze the constitutional issues -- it was unfortunately written by a very revered justice famous for the first treatise on American Constitutional law.

The concept was largely limited to the maritime and smuggling context for a fair amount of time, but was eventually extended beyond that in cases involving moonshine.

By the time the 20th century rolled around, there was this long line of cases. Justices, whether politically conservative or liberal, tend to be dispositionally conservative. Every time it's come up it's been reaffirmed largely on the basis of pure precedent, without any analysis from first principles.

Here's one of the most liberal justices in history (Brennan) discussing civil forfeiture:

At common law the value of an inanimate object directly or indirectly causing the accidental death of a [416 U.S. 663, 681] King's subject was forfeited to the Crown as a deodand.16 The origins of the deodand are traceable to Biblical17 and pre-Judeo-Christian practices, which reflected the view that the instrument of death was accused and that religious expiation was required. See O. Holmes, The Common Law, c. 1 (1881). The value of the instrument was forfeited to the King, in the belief that the King would provide the money for Masses to be said for the good of the dead man's soul, or insure that the deodand was put to charitable uses. 1 W. Blackstone, Commentaries.18 When application of the deodand to religious or eleemosynary purposes ceased, and the deodand became a source of Crown revenue, the institution was justified as a penalty for carelessness.

...

Decisions reaching the same conclusion have continued into this century. In Goldsmith-Grant Co. v. United States, 254 U.S. 505 (1921), it was held that the federal tax-fraud forfeiture statute did not deprive an innocent owner of his property in violation of the Fifth Amendment. There, the claimant was a conditional vendor of a taxicab that had been used in the removal and concealment of distilled spirits upon which the federal tax was unpaid. Although recognizing that arguments against the application of the statute to cover an innocent owner were not without force, the Court rejected them, saying: "In breaches of revenue provisions some forms of property are facilities, and therefore it may be said, that Congress interposes the care and responsibility [416 U.S. 663, 686] of their owners in aid of the prohibitions of the law and its punitive provisions, by ascribing to the property a certain personality, a power of complicity and guilt in the wrong. In such case there is some analogy to the law of deodand by which a personal chattel that was the immediate cause of the death of any reasonable creature was forfeited. To the superstitious reason to which the rule was ascribed, Blackstone adds `that such misfortunes are in part owing to the negligence of the owner, and therefore he is properly punished by such forfeiture.' . . . "But whether the reason for [the forfeiture] be artificial or real, it is too firmly fixed in the punitive and remedial jurisprudence of the country to be now displaced." Id., at 510-511.

Calero-Toledo v. Pearson Yacht Leasing Co. 416 U.S. 663 (1974)

One could be forgiven for thinking that he's reading a Scalia opinion instead of a Brennan opinion!