There's one thing to keep in mind about the original Wickard decision: it was made in an era where the courts were still mostly practicing the doctrine of "judicial deference".
What this means is that when there's a case where the courts have to rule on constitutionality of some law, they err on the side of ruling it constitutional, if there's any way to twist the wording such that it would allow the law - even if the legislators didn't think of that originally. The idea is that the laws enacted by duly elected legislators represent the will of the majority, which is sacrosanct in a true democracy, while the courts are not elected.
Thus, the balance has to be in favor of the elected institution, and the courts have a duty to try to "save" any law by coming up with creative interpretations of the limiting clauses of the constitution. Oliver Wendell Holmes, the Supreme Justice of the Progressive Era (when that concept was at its peak), summed it up as, "a law should be called good if it reflects the will of the dominant forces of the community, even if it will take us to hell".
That doctrine has been on the decline since the Civil Rights era, where SCOTUS was striking down state laws left and right as unconstitutional. But it still surfaces occasionally - e.g. in the recent decision on the constitutionality of the Affordable Care Act, and specifically its provision that requires people without insurance to pay a fine (https://en.wikipedia.org/wiki/National_Federation_of_Indepen...). The court said that the fine is a kind of tax, even though the text of the law uses the term "penalty" consistently, and the legislators that originally wrote the bill explicitly stated that it's not a tax back when it was under discussion (and its opponents insisted that it was a tax, ironically)
So that doctrine kinda explains why they were willing to come up with such an obviously contorted interpretation. What's more interesting is why it stayed with us for so long...
Other than that, you aren't missing anything. A significant chunk of federal laws since that decision was based in part or in whole on that interpretation. There is exactly one court case where that power was found insufficient:
https://en.wikipedia.org/wiki/United_States_v._Lopez
"The government's principal argument was that the possession of a firearm in an educational environment would most likely lead to a violent crime, which in turn would affect the general economic condition in two ways. First, because violent crime causes harm and creates expense, it raises insurance costs, which are spread throughout the economy; and second, by limiting the willingness to travel in the area perceived to be unsafe. The government also argued that the presence of firearms within a school would be seen as dangerous, resulting in students' being scared and disturbed; this would, in turn, inhibit learning; and this, in turn, would lead to a weaker national economy since education is clearly a crucial element of the nation's financial health."
SCOTUS ruled that it was a connection that is too remote. But even then, the ruling was 5-4 - in other words, 4 judges believed that the rationale above is sufficient to accommodate the Commerce Clause!
The problem with getting rid of this ruling is political. Since there's so much federal legislation that is only possible because of it, pretty much every political force (other than perhaps libertarians) has a stake in the game. For the liberals, if Filburn goes away, then so do important parts of the federal Civil Rights Acts (per https://en.wikipedia.org/wiki/Katzenbach_v._McClung), which prohibit discrimination of customers and employees on the basis of race, gender, religion etc. For many conservatives, overturning Filburn is undesirable because it would gut most federal drug laws (per https://en.wikipedia.org/wiki/Gonzales_v._Raich).