Yes, a given funnel cloud has a limited destruction band, but even that can be considerable. Note that the largest storms ("wedge tornados
) have recorded funnel diameters in excess of 1 mile in diameter* (1.6 km for the rest of the world).
Additionally, once tornadic conditions are encountered, the risk itself is present over a large area, and people across 100s or 1,000s of square miles (that's rougly 260 -- 2,600 km^2) may be cautioned to take cover, affected by other strong winds, by windblown debris, and/or local or regional damage caused by the storms themselves. High winds sufficient to be damaging can exist well outside the vortex, the condensation funnel does not indicate the limit of tornadic winds, and the entire storm may be rain-wrapped, preventing visual observation.
It's a bit like being in a crowd whilst a gunman is actively shooting. The damage cross-section of any one round is small. The net effect on an aware crowd is large. The potential for damage itself changes behaviour considerably and has a marked psychological impact.
Wind damage increases with the square of the speed, and gusts are not sustained winds. The damage caused by a 500 MPH tornado field experienced for five several minutes is thousands of times greater than a few seconds of 50 mph gust, and that gust alone could topple a tree. The tornado will pick up the tree and use it to wipe your house from the Earth, with earnest effort and taking its time in doing that.