You're asking a reasonable question.
I'm not sure how Rosetta was run; it had a longer germination time than many missions.
But for a typical science-driven mission, there is a thing called a "Science Traceability Matrix", or STM, that has science goals down one side and measurements down the other side. Every science goal must be traced to one or more measurements that will achieve it, and every measurement must correspond to some goal. If the linkage is not clear, the measurement (i.e., instrument - one spacecraft typically has many instruments) will be booted.
For more: http://csc.caltech.edu/references/Grogan%20STM.pdf [page ~8 has a matrix]
The science goals, in turn, are arrived at through National Academies studies, typically Decadal Surveys, done once per 10 years (often with a midpoint course correction). Membership in the NAS is a very big deal that few scientists achieve; the expertise of these people is unreal. That is:
state of science knowledge ->
decadal survey ->
science goals ->
traceability matrix ->
specific instrument choices
This is idealized, but mostly the process follows these rules. A particularly good example is Earth observing satellites, which are governed by decadal surveys like this one:
http://www.nap.edu/catalog.php?record_id=13405
One of the outcomes of all the space funding over the years is that there is now a somewhat orderly process for attacking big questions systematically.