Sure the hybrid has parents wildly different genetically. But if two individuals who have a high chance of producing fit offspring may by chance produce unfit individuals, then we also agree the hybrid would have to have been a minimally fit individual by chance born from two parents who have a very low chance of doing so. A person with Down syndrome is not a catastrophe of nature, and not fundamentally a disease, such that through successive backcrossing (women with Down syndrome are usually fertile) it could theoretically produce a different kind of Homo that would be fit for some imaginary environment conditions. Or at least would still be a far more intelligent creature than all other animals we have on this planet.