It's a long time since I read it, but there's a book called "The Structure and Interpretation of Quantum Mechanics" [1] by R. I. G. Hughes. The "Structure" part of it begins by building up most of the mathematical framework (including use of complex numbers, Hilbert spaces, operators, etc), motivated only by the desire to build a physical theory that is probabilistic in nature. It then shows how you can add one extra ingredient that turns the framework into that used for quantum mechanics [2]. I assume that everything discussed up to that point applies equally to Koopman-von Neumann.

It's a really nice book, very self-contained. I think anyone with a basic mathematical education (A-Level or equivalent) could get through it without having to read other things to acquire prerequisites, though they should be prepared to think quite hard.

1. The resemblance to the titles of Gerald Jay Sussman's "Structure and Interpretation" books appears to be coincidental. The title is meant literally: the book is split into two sections, one on the (mathematical) structure of QM and one on its (philosophical) interpretation. There are no similarities in style, pedagogy or subject matter to Sussmann's books and no use of, or reference to, programming. The author was a professor of philosophy at the University of South Carolina.

2. He actually lists a collection of alternatives for that extra ingredient, any one of which has the same effect when added.