Hardwood in the USA is mostly farm grown, so I'm not concerned.
I'm far from an expert, but I am in the process of planning our hardwood harvest in Vermont. The northeastern US produces a lot of hardwood and my impression is that very little of it is "farmed".
Can you point to sources? My attempt at searching didn't produce much either way. If you are right, I'd love to update my false impression. Or are you possibly confusing "hardwood" with "softwood", such as pine?
Last time I bought some pine lumber for a project at Home Depot there was a bunch from New Zealand and a bunch from USA, the stuff from New Zealand was cut to much better precision.