As I understand it, Markdown syntax was inspired by email syntax, so we have > for block quotations, * for unordered list items, etc. By this reasoning, a single newline may just be the email client wrapping the text, whereas a double newline means the writer is starting a new paragraph.
Initially I wrote it against the Daring Fireball spec but switched it over to track the output of the showdown javascript library.
Markdown is not a great language to implement. I wrote a hand parser because there are loads of places where you have to look ahead more than one token (not that I am a great lexer/parser expert).
This rule is quirky -- sometimes you'll end up with (for example) bullet points merged into a paragraph through this quirk.