Ask HN: How does a developer get started with usability testing?

I launched a web app a while back that got a good number of sign ups but all of the feedback I got from my users was the same: Usability really needs some work.

I decided to do things right. I did some research on usability and started over with regard to design. I've come up with some mock ups a la balsamiq that I feel makes it really easy for users to do what they need to do on the site.

Being a developer, though, I'd like to test this. How does one do so? I'm looking for more than just "go to a coffee shop and offer to buy them coffee herp derp." There are more facets to it that are question marks in my head like: What questions should I ask? How should I observe them? Should I have a plan going in to it? If so, is there a format I should follow?

If anyone here has some knowledge and/or resources they can share please let me know!