Some random issues of the top of my head..
1) Tiny tolerance/alignment differences between the target mount and adapted mount can cause the lens to become decentered or be mounted at a tiny angle to the sensor plane. 2) special mirrorless camera features might not work as well e.g. eye AF in sony / object tracking / etc 3) possible autofocus hunting issues on outer focus points or autofocus performance will be impacted 4) pro level lenses like the canon 500 F/4 combined with a 1Dx type body unlocks a faster focusing mode which wont be available with an adapted lens.
It is certainly possible to make an accurate mount adapter; see the Pentax m42 → K mount adapter. The "problem" is people don't want to pay for a good one; the Pentax adapter costs about 2x as much as the adapters everyone seems to buy.