I barely use anything sophisticated. When designing a complex system I usually start with a simple document that lists the use cases (this is the most important), system components, and some implementation details. There are a couple of architectural models I've tried using (such as http://en.wikipedia.org/wiki/4%2B1_architectural_view_model), but I haven't found them very useful in practice.

Instead of over-designing and engineering from the very beginning, I prefer a more iterative approach. Admit to yourself that the first time you design/implement a system you will most likely get a lot of things wrong. Get a proof of concept out there, and try to learn from it. The second time you design that system you will probably try to incorporate everything you know and are prone to overengineering and feature bloat (http://en.wikipedia.org/wiki/Second-system_effect). So, be careful.

Check out "The Mythical Man-Month". It's a timeliness book on software engineering.