Oh, they have thought about it, and very early on at that. When Jefferson was drafting the Declaration of Independence, it had this bit in it:
"He [king] has waged cruel war against human nature itself, violating its most sacred rights of life and liberty in the persons of a distant people who never offended him, captivating & carrying them into slavery in another hemisphere or to incur miserable death in their transportation thither. This piratical warfare, the opprobrium of infidel powers, is the warfare of the Christian King of Great Britain. Determined to keep open a market where Men should be bought & sold, he has prostituted his negative for suppressing every legislative attempt to prohibit or restrain this execrable commerce."
This was removed from the final version during committee review. Much later, Jefferson explained what he believed the reason for this was:
"The clause...reprobating the enslaving the inhabitants of Africa, was struck out in compliance to South Carolina and Georgia, who had never attempted to restrain the importation of slaves, and who on the contrary still wished to continue it. Our Northern brethren also I believe felt a little tender under these censures; for tho' their people have very few slaves themselves, yet they had been pretty considerable carriers of them to others."
The supreme irony here, of course, is that Jefferson himself was a slave owner until his death. And he had something to say on that, as well; written in 1820, on the occasion of the Missouri Compromise:
"... as it is, we have the wolf by the ear, and we can neither hold him, nor safely let him go. Justice is in one scale, and self-preservation in the other."