First of all, IANAL and I have no real-world experience with your scenario either. Below is all armchair reasoning. That said, my chair is pretty comfortable and I reason a lot from it.
Using patented technology without a license from the patent holder is illegal. On the other hand, you can't be sued unless people know you're doing something wrong. In-house technology that runs afoul of patents is therefore pretty safe, provided all the employees can be trusted... The moment you start exporting your technology, though, you're in trouble.
Making software open source does unfortunately not "avoid the hassle of patents". Whether your software is proprietary or open source, it can't use patents without permission. Conversely, you can hold patents on software distributed as open source -- although there would be precious little point to that, but maybe someone with an original licensing idea can correct me on that. In any case, a lot of licenses (most prominently the GPL) simply forbid this construction: if you release the software as "open", you implicitly grant everyone a license to use whatever patented technology you've included. Obviously, this can only apply to your own patents or patents you're allowed to re-license.
So: if you don't intend to publicize and you're worried about patents -- don't publicize. Conversely, if you do intend to go public, invest in good legal help first that specializes in IP. Checking your product for patents is hard, as is drawing up strategies for dealing with claimants. Nothing would be more disheartening than pouring a lot of effort in a product you will not be able to ship because a third party asks an exorbitant price for their patent use or refuses to license outright. They might buy your product from you, but that doesn't give you a startup.
This is not a question of "manning up", really (it's not like you can study real hard and then you've worked it out), it's one of economics. I have no idea how expensive the patenting business is, but TANSTAAFL in any case.