Right. The difference lies in the way that trust is achieved. The "I trust X because Y trusts X" transitivity is very easy to jump on, but often disastrous; just ask Bernie Madoff's investors. If you read in the newspaper that "X is one of the foremost scientists in his field", it is also dangerous to rely on just this pronouncement, and blindly trust X's assertions.
The other way for the layperson to judge specialists is to learn enough about their field in order to evaluate their predictions. Granted, this involves much, much more effort than lazily following the reputation talk. But it also makes it much safer to trust the people involved, and to maybe stop trusting them if they go astray at some point.
To bring forth examples, evolution is quite easy to verify after reading a bit of biology. For example, DNA analysis comes in a full century after the "Origin of Species" and shows us how amazingly related all living things are, deep within; thus giving precise shapes to the "trees of life" that Darwin first sketched in his notebooks. And there are many, many other predictions of the theory of evolution that can be easily checked by the unprejudiced amateur. You'll rarely find articles that say "Dawkins believes in evolution, and Dawkins is a great scientist, therefore evolution is true".
On the other hand, things like climate science seem (to me at least) to rely much more on the argument from authority for gaining lay support. If some prominent climate scientist had predicted in the 90's that "average temperature in the 2000-2010 decade, as indicated by methodology X, will be 1.4 plus or minus 0.1 degrees Celsius warmer than in the 1990s", I would have a much easier time trusting his or her claims now. But things seem to be moving in the opposite direction with this field, e.g. as witnessed by the very replacement of "global warming" (somewhat verifiable) with "climate change" (unfalsifiable), and the popular attribution of all kinds of meteorological phenomena to this cause, in the same non-comprehending way that, not so long ago, they were ascribed to the god of thunder.