>But when did the idea of slowly floating towards your fortified, gun-toting enemy in a giant gas filled balloon seem a good idea?

You don't fly in the balloon your soldiers do. That is what makes it as good an idea as trying to climb to the battlements whilst having rocks, hot pitch, arrows and sword blows rain down on one from above.

Even if the balloon simply diverts attention from your siege ladders then it might give enough advantage, it could also probably be used at distance to gain information vital to a successful siege - can you starve them out quickly, how strong are the numbers, where are cannon or arsenals located, etc..