Sounds harsh? Well non-refundable deposits are a thing in real estate. For example in my country it is common to put 0.25% down to take the property off the market, non refundable, allowing you to do due diligence and then 19.75% deposit within 5 days non-refundable, with a closing period of say 30 days to pony up the other 80%, usually from finance. Probably the 19.75% is lost if finance doesn't come through but IANAL not sure if that holds up court or not or if you can get it back in some circumstances. But people understand it's serious you might lose it.
If you don't really need to sell but wouldn't mind $5m this would be a good way to weed out non-serious people. If everyone says NO then that's OK you keep on running the business as before. If anyone says yes then you know they are serious. They might not buy, but you get $50k compensation, which might be less than your costs, but the point is the $50k is like a way of communicating information 'we are serious' than an actual payment.
Asking for $50k from someone who invests $5m is like asking $50 deposit for someone who wants to book a luxury hotel room for a week. Maybe I should have said $100k (!)