> I always thought that the shortest day shouldn't be the first day of winter, but rather the middle
That's not just you. Every traditional calendar says the same thing.
As far as I can tell, the idea that winter starts on the solstice, when it's been winter for more than a month (unless you're in the tropics), is an invention of Hallmark or some other calendar manufacturer. It is based on nothing.
I tried reading Poor Richard's Almanac[1] to see if it indicated the "beginning of [any season]", but if it does I didn't understand. (There is a column of "remarkable days", but it is often difficult to understand the entries in that column.)
Careful track is kept of the exact time of sunrise and sunset, and of what kind of weather to expect ("rain"; "snow"), but the seasons don't seem to be labeled explicitly.
Interestingly, I browsed through the Almanac of 1748 as a sanity check while writing this comment, and the sunrise and sunset times clearly identify June 14 (well, the 8-day period from June 10 to June 17) as the summer solstice (we'd expect June 20 or 21) and, much more weirdly, March 8 (exactly) as the vernal equinox, where we'd expect March 20. The autumnal equinox is September 11 and the winter solstice is sometime between December 3 and December 18.
Either Benjamin Franklin was terrible at calculating astronomical dates, or there was a calendar adjustment between then and now.
[1] http://www.rarebookroom.org/Franklin.html