Popular solutions:
* RabbitMQ
* Service Bus
* Kafka
RabbitMQ would be well suited, you define a producer and can then spin up as many consumers as you would like, each consuming from the same queue.
Preventing duplicate queuing should be done on the producer before it is queued.
It depends on the nature of the scaling and how much durability you want though, you may wish to simply maintain an atomic queue of work to be done, in which case any thread-safe list would suffice as long as changes were done atomically.
10k isn't that much in the grand scheme of things, a simple database could easily store such a queue if you didn't have that many nodes trying to consume the same table at once.
You would need to write stored procedures for transactional read & delete and inserts to prevent duplication of jobs.
In that case something like Redis might be good, which itself can also act as pub/sub and used for messaging.
Would you look to scale up more consumers as the queue lengthened, or would there still be a fixed number of consumers?