Yes, there was a referendum – but it wasn't a good one. It was advisory only – i.e. not binding – there was a narrow margin, no minimum turnout or majority requirements, and no concrete plan which was being voted on, or even any set of proposals from government. Two of the four countries in the UK voted to remain in the EU.
It think it's wholly reasonable to question a decision of such huge magnitude being made on the basis of a referendum in those circumstances.