No. Not if an observation leads to a testable, falsifiable theory, if that theory makes new predictions, if the predictions are verified, and if the theory in all its manifestations remains potentially falsifiable by new evidence. Like quantum theory, the best scientific theory we have, a theory that makes many other fields look like stamp collecting, and a theory that relies on observations of real-world phenomena.
As to cosmology, there are a number of cosmological theories based on observation that have led to testable theories, and those theories have survived reality-testing and have made predictions that are later verified in observations, like the recent observations of the CMB that support inflationary theory and led to some recent Nobel Prizes.
> I suspect most people would find that a somewhat extreme position.
Science is not a popularity contest, and it is not politics.
> Also, fraud is an issue in all fields of Science, so that seems somewhat an unfair issue to introduce.
Many people don't understand that some pseudoscientific fields are all fraud, in which case arguing that all fields have some fraud misses the point.