It would indeed defeat the purpose of going there. I have been to conferences before, but never alone. Usually, the person I went with was pretty social and I had no problems getting in touch with other people, through my colleague.
The worst case scenario doesn't sound too bad. I just don't want to come back without having spoken to anyone.. I would see that a bit as a failure, considering I'm going there to challenge myself.