And very few people had access to resources like the TX-2, the 2nd generation of the transistorized Whirlwind, the computer that defined how we build them to this day.
With a whopping 256 KiB of memory (64 Ki of 36 bit words); not many systems could afford that much memory in the '60s. Part of the MIT style of doing things is to have a large uniform address space, e.g. later the AI Lab got DARPA to fund a 1 MiB memory bank for their first generation KA PDP-10, that's one full address space, 18 bits of 36 bit words (a cons cell was a word treating each 18 bit half as a pointer). I was told that was considered to be beyond the state of the art back then, and this account strongly supports that: http://ljkrakauer.com/LJK/60s/moby.htm And, yeah, I can see the 4 extra bits per word theoretically available in case one plane failed inspiring Greenblatt, "the father of the Lisp Machine" and company to do a tagged architecture. (When I showed up in 1979, I was told MACLISP for the PDP-10/Decsystem-20 used the "big bag of pages" GC method, every page had only one type of object in it; no cache except on KL processors, so originally no locality penalty.)
I suspect Smalltalk has a larger object size granularity than Lisp in part because it was developed to run on systems with multiple banks of 64 K of 16 bit memory (up to 4; the Xerox Alto). The cost of bank switching is lower the larger the object to be addressed or swapped in (note, this is my personal theory, I have nothing to back it up except my study of these systems in the early '80s and intuition).