Many of the houses and buildings there are sturdier than you'd think. My parents house is at the end of a row of six or so that share walls. So they buffer each other's flanks during a hurricane. Their windows don't have glass. Instead it's screens covered by storm shutters with iron bars over those. Also, the houses are constructed of thick reinforced concrete rather than the picturesque wood, siding, and shingles that you might be thinking of. It's actually rather bunker like. Hurricanes can hit several times per year and those houses are built to take the punishment.
(Got word from my parents last night that the power was out and their street is filled with windblown debris that they and their neighbors will be cleaning up today, but otherwise they came through fine.)
However, its the destruction and risk that it could be WEEKS before help comes if infrastructure is completely destroyed... and an island with limited resources means that the people still on the island will (compared to mainland) very quickly start to realize that they need to eat and acquire things to survive... and survival mode is the worst part of any natural disaster. The damage done by Katrina to New Orleans was horrible, but honestly, the damage done afterwards by the people that lived there (that was done to survive, im not criticizing, morality changes real quick in survival mode) wsa pretty horrible, too. Imagine a small-ish island with not a lot of natural resources and no quick way to get truckloads of supplies en mass...
Not saying thats what's going to happen - but thats the kind of outcome people forget when they volunteer to ride out a natural disaster. The actual event is just the prelude to a pretty bad period of time for the area it hit... and the pain you will feel after the hurricane is likely to be much higher than the hurricane itself.