Computers had much less resources back then, and NCP had reached it's limits in terms of scalability. Running both TCP and NCP would have hindered progress and innovation. So the decision to outright deprecate NCP was necessary to keep scaling the network up.

Keeping IPv4 alive today doesn't hinder progress, and with the high bandwidth available nowadays the minor overhead improvements of IPv6 will be barely noticeable for the average user. Also keep in mind that the average internet user back in 1983 were a bit different from today ;-)

edit: typo