There was a in-depth article from Vanity Fair from 2012, "Microsoft’s Lost Decade" which tries to put together what happened.
http://www.vanityfair.com/news/business/2012/08/microsoft-lo...
Some excerpts:
By 2002 the by-product of bureaucracy—brutal corporate politics—had reared its head at Microsoft. And, current and former executives said, each year the intensity and destructiveness of the game playing grew worse as employees struggled to beat out their co-workers for promotions, bonuses, or just survival.
Then, in June 2004, Steve Jobs announced that Apple was releasing its new operating system, called “Tiger.” And inside Microsoft, jaws dropped. Tiger did much of what was planned for Longhorn—except that it worked.
E-mails flew around Microsoft, expressing dismay about the quality of Tiger. To executives’ disbelief, it contained functional equivalents of Avalon and WinFS.
“It was fucking amazing,” wrote Lenn Pryor, part of the Longhorn team. “It is like I just got a free pass to Longhorn land today.”
Vic Gundotra, another member of the group, tried out Tiger. “Their Avalon competitor (core video, core image) was hot,” he wrote. “I have the cool widgets (dashboard) running on my MAC right now with all the effects [Jobs] showed on stage. I’ve had no crashes in 5 hours.”
Longhorn was doomed. A few months later, Allchin brought together the Longhorn team and made the announcement: Microsoft couldn’t complete Windows Vista in time to hit the latest planned release date. In fact, the company couldn’t foresee any launch date. So a decision had been made at the most senior reaches of Microsoft: after three years of work, throw everything out and start over. It was decided, at least for now, to drop or modify many of the original objectives; no more using C#, abandon WinFS, and revise Avalon.
Apple was already in the market with those features; Microsoft was basically giving up in its effort to figure out how to make them work.