I have had similar experiences, more than once, but it wasn't awkward. Most of the time, people just say OK, and go on with their life. A couple times, we talked about it, and they commented that they wouldn't be able to get in touch with me again. We talked about whether being connected on Facebook vs. never talking again really would amount to much of a difference in the long run, and agreed that it really would not.

At the end of the day, a Facebook connection from someone that you otherwise don't communicate with, or run into in the real world isn't important. So those conversations are only awkward if you make them so.