In many other fields of professional work -- which laugh in the general direction of $200 per project -- spec work is an unremarkable part of the customer acquisition dance. I need an accountant who understands how to legally minimize tax burdens of an IP based business with service components with activities split across Japan, the US, and the EU. I was interviewing two accountants to decide who I would use. Both of them a) reviewed 3 years of tax returns, b) spoke to me for several hours, and c) broke out the books and looked at tax treaties. That is exactly indistinguishable from what the winner will eventually charge me for. But that time isn't billable. Cost of doing business.
Similarly, many of my consulting engagements get sold on the basis of me going to a client's office, getting introduced to their CEO, lead tech guy, and head of marketing, and then talking for between 1 and 6 hours regarding plots I've hatched in advance, while everyone in the room takes notes. If there were already a contract in place, a) that presentation would be a deliverable and b) that meeting would be billable, along with the rest of the week. At the end of the meeting, sometimes the CEO says "Great stuff. Let's think about it and circle back on this later.", which occasionally turns into a firm "No." three months later. This is, to plagiarize Thomas, life in the big leagues.
Ways of compensating for this: a) Charge. More. , b) Minimum projects of sufficiently large size such that customer acquisition is just a cost of doing business., and c) if you can't reliably explain to clients why you're better than a teenager at 1/20th the price either you're looking for the wrong clients or you're in the wrong business.