If I had to choose a AAA studio/publisher over alternatives, would go with Firaxis, makers of Civilization. An Indie developer though would be the best bet to getting a game that isn't made for just casual gamers.
Kerbal Space Program[1] has a some of what the OP might want, but with a more narrow, realistic scope. Also allows modding for extending the game.
Also Civilization 4 had some great mods (including several Star Trek ones)[2]. They were good enough, you actually felt you were playing a turn based space sim and not one on a planet. Not sure if any are out for Civ5 yet. I own it, but I have not had time in a while to to play much.
[1] http://store.steampowered.com/app/220200/?snr=1_5_9__205