Remember to separate the JVM from the Java language. Java the language is getting old - its not that Java is bad, its that languages like Ruby & Scala are better - programming languages have evolved since the early 90s.
A startup has to do all that you said, plus one more thing: dont reinvent the wheel. With languages like JRuby and Scala built on the JVM you can reap the benefits of the agile philosophy and still make use of the available solid java libs (Apache Commons comes to mind), and not have to reinvent the wheel, saving the startup a ton of time.