For one, astronomical systems are extremely regular. The same thing will happen, over and over, almost the exact same way, millions or billions of times before anything appreciably changes. It's extremely rare to find a system that is in the midst of significant change. We do find them, of course, and study them, but we do so after passing over countless "boring" areas of the sky.
This regularity means that anything we look at is extremely likely to be in a very stable configuration, and we can use our knowledge of orbital mechanics to rule out all sorts of things like weird dust clouds orbiting the star, which would not be stable (or would look very different if they were). Realize also that we stare at these systems (or at least check up on them periodically) for years at a time -- in this case they said 2006 to 2008, and again 2011 to 2012. So we can easily rule out some random thing passing between our telescope and the star precisely every 6.25 days for 3 years. It must be something stably orbiting the star.
We can also use the exact shape of the "light profile" to be pretty certain about the shape of the object passing in front of it. Over dozens or hundreds of observations, we see the same characteristic dip in light at the exact same period every time, and it matches the shape of a circle passing in front of another circle. It doesn't happen immediately, because the planet spends a bit of time only partly obscuring the star, and we can use this to discern that it is roughly circular in shape. We use this profile and the amount of reduction to calculate the size of the planet. Asteroids would be far too small to detect (we can't even detect all the asteroids in our own solar system!)
Also realize that when they say "light", they're usually looking at a whole spectrum of light. It's not one number that dips, it's a whole waveform that subtly changes. A dust cloud would be partly transparent, differently for different wavelengths, and change the light in a different way. In the extreme cases we can even use this to guess whether the planet might have an atmosphere, because it changes the waveform differently at its edges.
Space is also really empty. A bird might pass in front of your telescope on land, but in space, it's almost certain that nothing is going to randomly pass between you and what you're looking at, even over years.
And finally, even with all this, they spend years analyzing the data before they can be confident enough to make claims like this. There is a lot of data and a lot of time spent on it, and it's the culmination of many different lines of evidence. It's certainly not "flimsy".