Yes and no. There's an important distinction between _having_ something and _owning_ something. You can have possession of figures carved from contraband elephant ivory. But if anyone who was inclined to care about such things decided to take action, a court could compel you to destroy it. Same thing with any illegal item.
But these are all "ifs and buts." Hence why it's a matter of principle and not practice. At the end of the day, most people are fine with "buying" a book that they don't own as evidenced by the sales.