You make a good point, but it really depends on the content of the email. For stuff that's mostly text, without much in the way of interesting links, "open" rates can be deceptively low and you could potentially be losing active readers if you remove them.

On the other hand, your email could be being spamboxed (or worse, blackholed) because half of your list has marked you as spam, and you're continuing to send to them once a week.

There isn't really an easy one-size-fits-all solution to that. Using a decent email service provider is a good idea, if you have the budget for it. They'll be able to give you a lot more information about your email's performance, likely have better delivery rates than your current mailserver (assuming you pick a provider that doesn't suck) and, importantly, will be able to identify and remove recipients who have marked your email as spam.

Edit: Deleting bounces, on the other hand, is extremely important, particularly if you have lots of webmail addresses. Possibly the most important thing you can do to maintain or improve delivery rates. That really should have been higher up on the list!