Most of us draw the line around 5.8.6 or so, given that all major distributions include a Perl at least that new. In fact most of them are at 5.10+ now, with a few notable exceptions.
I would only aim to support 5.8.9, 5.10.1, and 5.12.2, and at this point I don't know of anyone still using 5.8, so I just stick with the two most modern versions.
When 5.14 is released, it will really be time to kiss 5.8 goodbye.
Could your audience also be limited by the fact that your modules are IRC bots and bindings to XML parsers that already have bindings? :)
Anyone who hasn't upgraded Perl 5 in eight and a half years seems unlikely to install new software.