Because unless the HTTP response is a redirect to the www. url (or vice versa) it's technically a document available at a distinct URL. For most servers, the two hostnames point to the same server (and vhost) but it's conceivable that they might not.

You've got a similar problem with arbitrary appended query strings.

I guess the clean way to handle this would be to detect very similar URLs and ask the user if it's the same document.

Technically, you can have one site on www.website.com and another on http://website.com and hence they can be considered different and unique.