Maybe later editions, but in the beginning it was troff (I just talked to Chris Espinosa about it the other day).

Too early, I think Framemaker came out in 1986.

The Wikipedia article about FrameMaker says it was first released in 1986, and for the Mac in 1990, five years after the original Inside Macintosh books were published.

FrameMaker was used for the new Inside Macintosh series, starting in 1992. From my copy of Inside Macintosh: Overview, on one of the last pages:

  THE APPLE PUBLISHING SYSTEM

  This Apple manual was written, edited,
  and composed on a desktop publishing
  system using Apple Macintosh
  computers and FrameMaker software.
  Proof pages were created on an Apple
  LaserWriter IIɴᴛⅹ printer. Final page
  negatives were output directly from text
  files on an AGFA ProSet 9800 imagesetter.
  Line art was created using
  Adobe™ Illustrator. PostScript™, the
  page-description language for the
  LaserWriter, was developed by Adobe
  Systems Incorporated.

The same page acknowledges, by name, the members of the writing, editorial, and production team.