I sometimes sigh at poorly performing apps. I've been sighing a lot at the feedly app lately. But that doesn't stop me using it, because the functionality is compelling enough.
Putting effort into performance is probably something you'll want to do at some stage if your app gets big enough. But a lot of the time you have more important things to do right now. Users will put up with an awful lot if you've got something they really want; get that part down first.