I don't really do ongoing payment (rental) schemes, out of pure personal preference. (If a customer asks for one, I can do that, though -- it's just not a default option). I don't like it when other software does this, so I'm not going to do it myself. That's not an economic or business-based decision and not a recommendation. I'm just mentioning it to explain why that's absent here.
I take a look at what my development costs were to develop the library, and use that to estimate what it would cost another shop to replicate the same functionality. I try to subtract out the portion of the development costs that are reflect the API design and other things that are unique to library development (including documentation).
Then, once I have an estimate for what it would cost my potential customers to develop the functionality, I cut that in half, and that becomes my asking price.
I also sell libraries in two forms: with source code and without. For those who want the source code (which is most of my customers), I charge double what the asking price I computed above is.