I think shift+enter for new line would be a better choice if you were to highjack an existing shortcut. Probably barely used by anyone and it would be consistent with a line break shortcut in word/powerpoint and elsewhere.
Committing a formula needs to be a simple shortcut because it has to be used by the least technical users. You don't want to create a "how to exit VIM" mess in a retail product.