Admittedly, it might be someone ELSE'S email address, but they could manually unsubscribe from your mailing list later.
That's half the point of verifying e-mail addresses. Doing this is likely to engender significant ill-will in anyone who is an unintended recipient of your e-mails.
---
You are receiving this email because someone (presumably you)
subscribed $(email) to the $(maillist-name). Subscription
request was received from a network address of $(ip) on $(date).
If you did not request this subscription, or would like to be
removed from the maillist for another reason, visit this link
to unsubscribe - $(link).
1) Enter email on web page.
2) Open confirmation link in email inbox.
If you skip second step - there is a risk that email ends up in spam folder and is constantly delivered there. Another risk - user did not really request your subscription. That would often result in your emails being marked as spam.
1. Before the near-universal requirement for confirmed opt in, it was a reasonably common attack/prank to sign up someone else for tons of lists, causing them to receive unmanageable amounts of email per day. Alternatively, sign them up for lists they'd consider offensive, or that'd get them in trouble at work. Confirmed opt-in stopped this attack.
2. We've (computer geeks, security experts, etc.) spent years trying to teach users that no, they really shouldn't click on that link in an unexpected email, even if it promises cute kittens. That continues to be a problem. I don't know how an expert, much less random Joe, could tell a real "click here to unsubscribe" link from a fake one, if you have no idea who the sender is. Researching if its a legitimate unsubscribe link—and not a link that'll confirm to a spammer that you're reading his email, or pull up a page full of the latest browser or Flash exploits, makes it far more expensive for the random person to handle an opt-out than it is for the person who wants the subscription to handle a confirm.
Confirm emails are easy for a random person to handle: just ignore them. You won't receive any more emails if you take no action.
3. Not all email addresses are nearly-free to send to. Well, they're all nearly free for the sender, but not the recipient. Some email addresses cost 20¢ per message (e.g., SMS gateway). Some email addresses page people, waking them at night. These addresses aren't always very different from other people's addresses (I've personally had misdirected mail to both types). Someone can also do this as an attack. Sending a single message is bad enough, but is the minimal harm you can do; but imagine if anyone could subscribe your text message address or pager number to linux-kernel.
4. Realistically, while some mailing lists are very easy to get off of (e.g., development list for a random open source project), many commercial lists aren't. I've had people give my email address (assumably by mistake) to the College Board on their PSAT (hundreds of emails per month, many from different senders, no way at all to stop them); sign me up for an Xbox account (took me at least half an hour to figure out how to stop that—-apparently you're supposed to phone Microsoft; thankfully I finally found another way); sign me up for various things at EA (nothing on their web site, contacting support was absolutely useless); etc.
5. Your ISP probably has policies against sending unsolicited bulk and/or commercial email. If you aren't confirming your subscriptions, you can expect your ISP to receive complaints, and you are probably in violation of their TOS.
Not sure how much this still happens, with spam being so much cheaper to send with botnets.
Secondly, if I can sign someone else up without their knowledge and get you to send then unsolicited mail, you're risking legal problems.