Nice post! What about the following (untested) version of the sentence function:

    (defn sentence [mapping]
        (->> :start
             (iterate #(pick-next-word mapping %))
             next
             (take-until #(= (last %) \.))
             (clojure.string/join " ")))

There is no loop, and it almost reads like English: "Using :start as a departure point, iteratively pick the next words until you find one that ends with a dot. Join all the words generated so far with spaces."

Small catch: take-until is not yet in clojure.core, however it's easy to add to your utility toolbelt and there's already an issue open for it http://dev.clojure.org/jira/browse/CLJ-1451