1. 50 (100 if you have reddit gold) random subreddits from your subscriptions (or from the default subreddits for logged-out users and ones that haven't customized their subscriptions at all) are selected. This set of selected subreddits will change every half hour, if you have more subscriptions than the 50/100 limit.
2. For each of those subreddits, take the #1 post, as long as it's less than a day old. Order these posts by their "hotness", and then these will be the first X submissions on your front page, where X is the number of subreddits that have a #1 post less than a day old. So you get the top post from each subreddit before seeing a second one from any individual subreddit.
3. The remaining submissions are ordered using a "normalizing" method that compares their scores to the score of the #1 post in the subreddit they're from. This makes it so that, for example, a post with 500 points in a subreddit where the top post has 1000 points is ranked the same as one with 5 points where the top has 10."
https://www.reddit.com/r/dataisbeautiful/comments/2lhhiu/the...
This is an old comment (2014), so it's about so-called "old design" (old.reddit.com). I don't know much about "new design", because I don't use it.
See also this interesting related discussion: https://news.ycombinator.com/item?id=8567494 (and also this: https://web.archive.org/web/20180528053032/http://www.kairay...)