Not necessarily. Black locust (Robinia psuedoacacia), e.g., is one of the faster growing hardwoods in North America, but produces very hard, dense, and strong wood.
Throw in some baobab genes to help redistribute the growth/weight.
good point, add in some density genes to make them hard as rocks
Or maybe not quite that hard. Some woods are already hard enough to be inconvenient to work. Though maybe it's still all benefit if you're just trying to make industrial feedstock for cross laminated timber or something.