There is always going to be an issue of trust at some point: the linux code base is public, but almost none of us compile linux and the packages we need to build a complete OS from source. We end-up getting pre-built packages from repositories that are often distributed all around the word. How easy/hard would it be to compromise some packages or some repositories if you are determined enough?

You certainly could not compromise a base as large as the number of Windows users, but you could target your efforts on distributions that have key infrastructure roles, like servers, routers, firewalls...

Another vector used to compromise free software is to participate in it. Paid agents can actively participate in open source projects and allow clever exploits that could pass as bugs if uncovered.