I think one serious issue was that they took way long to realize portability.
They had rough group implement MICA OS that was supposed to be partially compatible. But the dropped MICA and PRISM and instead adopted MIPS and Ultrix. But then they never bothered to port VMS onto MIPS and their workstation line.
Then they mismanaged their whole VAX line and massive lost out to the improving Unix offerings, partly for performance and partly for 'openness' reason.
I think if they were smart they would either have gone 'all-in' on MICA and make it the future of both Unix and VMS. Or simply support both Unix and VMS on all of their platform. They were big enough that this wasn't some incredibly complex task.
They ported VMS to Alpha, but that was a 64bit machine that most people didn't need. Meaning they lost in the lower spec server spaces where Sun and HP were selling tons of Unix servers.
Had they ported VMS to MIPS and also had sold lower spec server with all these features, VMS might not have become so niche.
Personally I really believe that their complete mishandling of RISC, despite having arguably the best chip design team in the industry to be the reason for much of their failure.