Who's "we"? - It doesn't apply to everyone in the world, so you're assuming some limitations in who you're referring to.
GP makes a fair point. If you mean by "pagan" simply non-Christian and non-Jewish, then to make it relevant to call it a pagan library you would need to establish that it was curated specifically to exclude Christain or Jewish themes. You might as well call it a "non-Mithraic library", if it happens to exclude mention of Mithras, which was also an up-and-coming cult among the Romans in the first century. Then it would be incorrect or presumptious to call it "non-Mithraic", unless you'd first established that it contained no mention of Mithras. And the only reason you'd do that is if Mithras held a particular parochial relevance to you. You understand that not everyone holds up an image of Mithras as a prism through which to view everything else.
OTOH, if you mean by "pagan" just that it's Roman, but from before Rome converted to Christianity, then just say it's a first century Roman library.