What about 'drag-to-hover' and 'tap-to-interact'? I don't feel like this is an unsolvable UI problem, when some core apps have 3-finger-pinch-and-drag-while-swiping gestures.
Worth trying. Startup idea: buy the rights of a critically-praised-comercially-failed adventure game from the 90s, port it to iPad with this mechanic. :)
I would play.
Currently play Broken Sword on pass-the-tablet with screen duplicate streaming to the wall size tv, for effectively social couch coop. Try it with friends and family, or whoever you happen to be locked in with at the moment.