Its only allowed to be mutli-family because its grandfathered in. Believe me, the town and neighbors would rather it be single-family, because multi-family brings in a lower class of occupant.
And heres the other thing. It was originally a single family house that was converted to multi family. So things change, markets change, demand changes.
Codifying usage in zoning laws and making it immutable might be a luxury affordable for places like NYC, but its not affordable for places with tons of sellers and little buyers, places where foreclosure rates threaten the value of homes with current mortgages.
The occupants of my house go to restaurants, rent kayaks, go to the local attractions. This is a tourist town, and vacation rentals are expanding our reach.
People will stay at a vacation rental that wouldnt stay at a hotel. Because they get a kitchen and a room for the kids. They will stay for a week in the summer. Who wants to stay for a week in a hotel room with four kids?
Vacation rentals are keeping this town alive, and I don't think this is being represented in this debate.
There is near unlimited affordable housing available in this area.