The problem with appealing to the Founding Fathers is that they were profoundly uninterested in equality of opportunity, due to who they were. It is a useful exercise to consider what they would have done — that does not itself demonstrate anything.
1. The Founding Fathers were products of their place and time. Many other countries engaged in the slave trade and employed slave labor, and women played a similar role in other societies. Racism and sexism were commonplace around the world during their time. You can reasonably argue that American slavery was uniquely cruel and long-lived if you want, although I would say that position downplays the horrors suffered by slaves under other governments.
2. The Founding Fathers, for all their faults, were clearly very poignant observers of human nature and history. They thoughtfully designed an extremely successful form of government that has inspired countries around the world to copy their model and arguably contributed to material prosperity unparalleled in history.
None of that is to discount what you're saying, but it's only one piece of the puzzle.
Slavery eventually eliminated (before most other nations), at tremendous cost not appreciated by many benefiting from that payment.
Colonial slavery oppressed conquered people's on their local soil where they were the majority. This is hard to keep up for a global power constantly at war with its neighbors.
American slavery oppressed ethnic minorities ripped from their homeland and taken to a different continent. Slaves had a harder time coordinating rebellions as they were not a single conquered peoples, and were in the minority so couldn't count on winning battles of attrition.
American slavery, like the Roman version it was patterned against, could have continued for hundreds of years if not for the Civil War.
Colonial slavery though was a lost battle from the start. Colonial powers needed to develop their colonies, but the moment you do that you increase the chance of successful rebelion. The only way to defuse it is to reduce the oppression.
The Founding Fathers didn't have a shared opinion on the issue, and the only concrete action they jointly took was directly protect it and to bake in extra political power for slavers.
> but had to compromise lest the fragile Union break before it could form
The people those founding fathers who wanted slavery eventually eliminated had to compromise with were also founding fathers.