It's the same concept as insurance: it's often better to take a guaranteed loss of X, than 10% chance of a loss of 10X -- or even 8X, due to a loss aversion.
So, let's say that you're in a group of 10 people and one of them will suffer an 8X loss, but you don't know who. You each set aside X, and pay 9X to the guy who turns out to take the loss. (Refund of contribution plus 8X.) You pay X for the overhead of managing the arrangement. Boom, you reinvented insurance.
There are cases where the evidence is strong, but neither side likes the uncertainty of leaving it up to the trial, so they make the above tradeoff, and save on the costs.
As for your question about "why not have a mini-trial anyway", the prosecution is required to disclose their evidence at the allocution, so there (some level) is transparency about the basis for guilt; but if you have to commit to the regular trial with all its procedures and protections, then that defeats most of the purpose.
Often, people confusedly moralize about plea bargains, when their real objection is that the government could credibly claim to be able to get an 8X penalty at all, which seems excessive given the facts of the case.
Also, not just an American thing: https://en.wikipedia.org/wiki/Plea_bargain#Use_in_civil_law_...