Working at the office is about "staying in touch". When you are in the office you see new directions the company is interested in long before you get an assignment.
It doesn't matter if you are in an office with the door closed, or working from home: either way you miss things that you should have been told directly but nobody thought to tell you. So you end up doing otherwise great work that it leading the project in the wrong direction.
That is when you are in the office someone from marketing can see you in the hallway and ask about the hot new feature of today without invoking all the formal channels to take to the guy remotely. Thus when you are working from home you lose a lot of opportunities to do what is needed as opposed to what you are told. People who can figure out the right thing and do that are more valuable than people who only do what they are told.
Note that the title is correct: all employees should be trusted to work remotely from time to time. However that does not mean you should work from home all the time. You should be a regular at the office so that you don't lose touch. At the first sign of illness (you or your kids), bad weather, or a hard no interruptions problem: you should be out of the office and working from home. However you should also be a regular at the office for the benefits of being at the office.
Of course this depends on your job. If you are a contractor you might be in a situation where you do the letter of the contract and are gone - the only time you might want to do something else is if you are trying to turn the contract into a full term position. Even then though, some contracts are get release N out the door, with a possibility of renewal for release N+1 - in that case you may have the option to change what is in release N.