That sounds very interesting. Do you have any links you could share with more details? I've been trying to come up with a way to get some friends involved to play out some of the scenes in a book I'm working on and those techniques might really help. The only thing I've seen that sounds related is a game called "Downfall" [1]. I think it's a little too structured to use for this purpose, but I liked the concept.
What's the genre of your book? I have a couple lighter-weight RPGs to offer as suggestions.
FATE - you have skills ranked as things like mediocre, expert, poor, etc. (don't recall exact terms). You roll dice to determine how well you do as modifiers (+/-/0) to your skill or attribute when appropriate. Fairly freeform character creation and play. Genreless. You can make it fit almost any genre.
Dread - This is horror focused. The mechanic is a Jenga tower. If a player wants to do a non-mundane task, they pull a block, if the tower falls, that character is out of the game (dead, went mad, whatever, you choose). Tension develops as the tower gets harder to pull from. Ebook is $10-20, can't recall, got it for free for buying the print book at a convention. Very lightweight, while horror focused it is otherwise independent from your setting. Create character sheets describing their history, their skills then are "what does it make sense for them to know". Does it make sense for the programmer to be a sharpshooter? Maybe not, maybe a difficult shot requires two pulls for them. OTOH, a soldier character may only need to pull if the shot effects the story more significantly. You can wing it, but be consistent with your players.
The classic example is Raymond Feist's Magician. There's a scene near the beginning where our characters choose what class they're playing, and then you can watch them level up throughout the rest of the book.
(Not that it's a bad book; I like it a great deal. But once you know what to look for, the signs are everywhere. It is fascinating to see his writing skill get better throughout. It's also interesting to read the original version with the Author's Preferred Edition, where he put back all the bits that his editor cut out, and realise exactly what editors are paid for. The original is so much better.)
Also a benefit of the lightweight systems for this as they usually don't have a notion of classes. Maybe an idea of competency in some fields instead.
Like Lord of the Rings and Star Wars?
Guilty pleasures, they are two webcomics that re-imagine those two works as role-playing campaigns. The Star Wars prequels actually make more sense that way.
And I also think the Greyhawk books started as a setting for a private game, then became books, and finally a published rpg setting?
Now, I haven't done this yet, but I want to take Fiasco and create less hostile games. The format is good for this. In your case, you could give people the characters from your story, give them a location and known relationships. Give them a goal (escape the cannibals, reach the town within a week before running out of food, etc.). Then let them play with the scenario. Your players would then be bringing the characters to life. Perhaps not directly mimicking the story that you had intended, because it's more freeform and player driven, but you can maybe get an idea for how your characters actually are as people.
Fiasco = Improv RPG. That's my way of describing it, at least. The rules as written are around tragic, horrible things happening to flawed people (literally, at the end of the game you roll your dice to determine your fate: worse than death, dead, in prison or similar, etc.; then you create a story to explain that fate, maybe you died in the game, maybe you died after when someone else came after you for revenge).
But I have a strong feeling that this could be converted into something less tragedy focused, if you can get player buy-in.
http://www.lamemage.com/microscope/
(we built a crazy setting somewhat reminiscent of "The Expanse" (independently) which we used for a bit of swashbuckling sci-fi - sort of "Firefly" meets "The Martian" meets "Ex Machina").
The writers of The Expanse based their books off of role-playing sessions they did. George R.R. Martin also was an avid role-player. I believe his Wildcards novels were based on his games in the RPG Superworld.
For some notes from our previous (and rather epic campaign) have a look at our GMs (now defunct) blog for the setting and role-playing in general:
http://thebookofworlds.blogspot.no/
Overview and links to more: http://thebookofworlds.blogspot.no/p/world-of-argos.html
The first post about the setting (Incidentally the character that broke from the group was mine, to much later surface as "the second witness", while Franco earned the title of "the first witness"):
http://thebookofworlds.blogspot.no/2009/08/boom-headshot.htm...