Previously they said that if the UK left the EU, and Scotland left the UK, they could take on the UK's EU membership. This seems to be what they're aiming for with saying they want the referendum before the Brexit process is complete.
That said, it's never been confirmed that would actually be allowed, and with the UK government likely to trigger Article 50 this month, I'm not sure the door is even open for that. And given the EU's fairly strict and slow entry process, as you say, Scotland could be left outside of both unions for many years. I'm not sure if they'd even satisfy the EU's economic requirements on their own.