"In The Fifty-Year Mission: The Next 25 Years oral history of Star Trek by Mark A. Altman and Edward Gross, DeForest Kelley explained that he was disappointed that Dr. Leonard McCoy and his crewmates only appeared in one scene, commenting: "When I read the script and saw we were only in the first ten minutes, I thought it was best to pass and go out with [Star Trek] VI." At that time, Kelley didn't know that Nimoy also said no and he admits, "I certainly wouldn't have done the film without him in it.""

I’ve seen accounts that the cost of his insurance was a factor in his rejecting the film (he wasn’t denied a role), but most accounts I’ve seen focus on his (and much of the TOS cast’s) issues with the structure of the story and the roles they would have had in it, and how they were much happier having The Undiscovered Country as their last film in their roles.

Like in 'Pitch Black'.