That is probably part of the argument that was made to require a parliamentary vote. Which is conceivably a fair judgement.
But on the other hand what is the purpose of having a referendum to leave the EU if it is not intended to be enacted in law? It would be rather nonsensical to hold referendums that have no effect and could be discarded at will.
Nevertheless, as we saw the substantial majority voted to honor the referendum and a fairly small minority did not. That leads me to believe that the overwhelming sentiment is to honor the democratic vote and the will of the people.
The risk of denying the democratic majority is the risk of the 'tyranny of elite minority'. Its hard to decide which is the true injustice in an abstract fashion. Its very much a question of context and specifics.