If Smalltalk and Lisp are so magical then why didn't they win?
That makes Smalltalk super easy and convenient to develop in, and very consistent across platforms. But it's a roadblock to publishing code, working as a team, deploying, and the other things you need to do with software that isn't a toy. The Smalltalk environment and community certainly have ways of doing all those things, but they can be clunky, and some of them are quite recent.
Meanwhile, C++ was a bit of a pain to write, but once you'd written it, sharing it was as easy as emailing a file, or checking it into RCS (or that hot new CVS thing), and building a production release was a matter of running 'make' and going for lunch.
Java kept C++'s simple approach. Smalltalkers at the time laughed at how caveman-like the experience was. But it turned out that that simplicity was exactly what you needed to grow virally on the then-nascent web.
I can't say anything about Smalltalk because I don't know enough about its history, but at this point in time it's pretty sure that unless some sort of major event happens, Lisp won't ever go mainstream.
It's not that simple. They lost for very pragmatic reasons: Smalltalk and Lisp were worse than Java in many ways (even smaller than Java's initial implementations, Smalltalk's binary images were undeployable in production, these two languages are dynamically typed, etc...).
"During the late 1980s to mid-1990s, Smalltalk environments -- including support, training and add-ons -- were sold by two competing organizations: ParcPlace Systems and Digitalk, (...) Both firms struggled to take Smalltalk mainstream due to Smalltalk's substantial memory needs, limited run-time performance, and initial lack of supported connectivity to SQL-based relational database servers. (...)"
"In 1995, ParcPlace and Digitalk merged into ParcPlace-Digitalk and then rebranded in 1997 as ObjectShare (...) The merged firm never managed to find an effective response to Java as to market positioning, and by 1997 its owners were looking to sell the business."
There was also Strongtalk, a strongly typed variant of Smalltalk. The guys who wrote Self and the guys who wrote Strongtalk were both eventually acqui-hired by Sun and they were the ones who wrote HotSpot [2][3]
At least the amount of insecure C code decreases, as it gets written in a safer language
"However, I believe that worse-is-better, even in its strawman form, has better survival characteristics than the-right-thing, and that the New Jersey approach when used for software is a better approach than the MIT approach."
"There is a final benefit to worse-is-better. Because a New Jersey language and system are not really powerful enough to build complex monolithic software, large systems must be designed to reuse components. Therefore, a tradition of integration springs up."
Richard Gabriel https://www.jwz.org/doc/worse-is-better.html
And now that we have good, modern high level languages with robust but flexible type systems I'd choose Haskell or Ocaml or maybe Swift over Smalltalk or Lisp if I wanted to stick my neck out technically on a project. I consider dynamically typed languages to be significantly "worse" in an absolute sense for building non-trivial software.
I don't know if there are any typed SmallTalks, but Common Lisp has optional static typing: you can declare types and the compiler will enforce them. This seems to me the best of both worlds: the freedom to interact with your program dynamically, with the safety of strong static types.
Even worse, Common Lisp will use your type annotations to optimize code, which can change the behavior of your program. It's better to think of them as type hints for optimization than to think of them as optional static types.
The type of analysis applied to Lisp is adapted to its type system: you have ranges of values, unions, etc. In particular SBCL can exploit some relationships between runtime values and types (I detailed it recently, so I prefer not to repeat myself: https://news.ycombinator.com/item?id=12222404)
As far as I am concerned, it is already quite powerful to detect the type-related errors that I tend to do. The part that cannot be decided during compilation is left to be checked at runtime, which is strong dynamic safety. Instead of manipulating untyped bits, you have all you need to never corrupt the state of your program. The good news is that if implementors make progress with type inference, you don't have to change the language: more checks can move from runtime to compile-time (the hard stuff would be the check the "satisfy" type ;-).
With some types left to be checked at runtime, you end up in a situation that is no worse than statically typed programs which throw exceptions for other kinds of errors. Sometimes, those errors arise because programmers implement a dynamic things, like ad-hoc tags to identify objects at runtime (so this is basically a dynamic type system in disguise). Other errors related to safety are tied to temporal issues, deadlocks, permissions, priority, security, etc. You can't always encode them with types. There are formal tools for dealing with them, but at this point, whether you use them generate safe code in Lisp or OCaml is not very important (and generating Lisp code is straightforward).
> Even worse, Common Lisp will use your type annotations to optimize code, which can change the behavior of your program.
This is a strange concern. The modified behavior belongs to the set of behaviors that would be possible without checking types, not a new, faulty behavior that contradicts what the code says.
This frees up an enormous number of brain cycles to design the upper layers.
It embraces it's host (JVM, JavaScript) which enables tremendous reach and code-reuse; at the same time it fixes most WATs and subtler pitfalls (by using immutable, persistent data (structures) by default).
'Reasonably sane' is a matter of opinion. I really, really dislike Closure. I don't want to ban anyone from using it, but it really feels wrong to me. Far better to just use Lisp on the JVM, with a well-defined interface to Java.