Awesome idea! But as a remote server, is the RasPi really up to it?
The GPU would be mostly idle in these cases, as Servers don't usually run a Graphical Interface, which would mean more RAM for the system.
All in all, this might well be an Application the RPI is more suited for than functioning as a media center.