In that case, you might as well just have numbers, since you can correlate numbers with colors, at that point.

I would rather see colors indicating the "heat" of a comment. That is, its numbers, in relation to other numbers in the thread. In other words, you wouldn't have the following:

Blue == 5+ Yellow == 10+ Orange == 20+ Red == 50+

Instead, you would have:

Blue == average score in this thread/topic Yellow == 5% higher score than other posts in this thread/topic Orange == 10% higher score than other posts in this thread/topic Red == 20% higher score than other posts in this thread/topic

This would help provide a more meaningful context for what posts are rising to the top in that particular conversation and the highest would always rise to the top. The top 20% of comments would always be the top 20% of comments, even if the lowest scored comment in the entire discussion is +10 points.