Postman founder here. Postman let's you run tests outside of Postman through the newman tool (free) [1]. Just take collections outside Postman and run them anywhere you want. Collections can have Javascript scripts embedded inside them for request chaining (transferring data from one request to another) or just writing integration tests. Tons of possibilities and we have some great new stuff coming up around this.

[1]: http://www.getpostman.com/docs/newman_intro

There are many better ways to do automatic testing. I've never used Paw for that, and I don't think it competes with e.g. postman on that front, if that's what you're asking.

However, for manual testing / having it write curl commands for you / making JSON really easy to work with by hand, it's fantastic, and I've converted a few people at work to it. Developer is also very responsive and nice to work with on Twitter.

Edit: you can also write extensions in JS for it. Not quite as accessible as e.g. Atom, but still decently hackable: https://luckymarmot.com/paw/extensions/

Try Runscope (disc: founder) for that. Paw is my favorite desktop app but we are purpose built for automated testing and monitoring.

As the sibling comment says, there are probably better tools for automated testing. However, I haven't dove too deeply into this aspect of Paw, but I know that at the very least there is a "Run All" shortcut which runs all requests you have configured. The sidebar shows which requests succeeded and which failed. There may be some way to automate this. (It apparently supports extensions, if this functionality isn't built-in.)