The one thing that will haunt you if you do not go native is the "feel" of you app, compared to the native apps on the same platforms. Most games solve this problem by having their own "immersive" GUI which often works because GUIs in games are fairly limited. Usually a responsive/Cordova type app will either select one of the native "styles" and mimic that, OR select to mimic a "website" type interaction, but regardless, it will never feel quite native on mobiles/tablets.
From another point of view, if you have to build everything yourself, going native means it's going to be a looong time until you cover all platforms (Android, iOS, responsive web). Depending on your app, doing cross platform first, THEN invest in native when you've got a hit under your belt.