Yes it's definitely a challenging undertaking, which is why no one has come up with such platform.

> One idea I had was to do some form of performance evaluation of each theme/plugin and show a score or impact grade to the user when they are enabling it

This would be good. You can also crowdsource part of that effort if it's time intensive.

* > I know a few people who have had problems with speed with their Wordpress sites, will tell them to check out Cachoid. *

Feel free to pass on my email (joe@). Thanks so much!