I've been thinking just in the last few days that it would be cool to write up how I did it.
The basic way is to create a document with nothing in it, then a second document with a very small difference. For example in Movie Magic I had one new document that I saved without putting anything in it, then a second where I entered the letter "A" into a certain field.
I made hex dumps of them both, then compared the hex.
Then I would make some modest changes and additions, such as the letter "B" in a different field, or I would change the "A" in that first field to "AB".
This approach only really works if you've had experience implementing file format code; it's particularly helpful to have designed original formats completely from scratch. For these specific reasons I am better than most at file format reverse-engineering, however I don't really have a clue about doing that for network protocols as the Samba folks did.
Once I have some guess about the file format, I start writing a file interpreter, typically to dump the binary format into human-readible text. "Actor's Name: A", "Movie Title: B".
Rather more importantly, that binary file dumper is chock-full of assertions. It's rather more important to get the assertions right than it is to display the actual file payload values.
Loosely speaking, a set of correctly-implemented assertions is, in itself, the documentation for the file format you just reverse-engineered.
It's also important to round-trip. So I come up with some very simple text input format, then a filter that creates documents that are readable by Zeni or Movie Magic, then I try to open the files. If that doesn't puke on my shoes, I edit the file a bit, then run that edited file through my human-readible dumping program.
Lather, rinse and repeat. It is very tedious and slow but it is quite cool when you discover something that works.
I've been puzzling over how I could offer a consulting service where I would reverse-engineer documents as a service to other companies. No doubt there is lots of need for this but I am unclear as to how to market it.
The DMCA only covers Digital Restrictions Management. If the DMCA doesn't apply, as it did not in the case of files that you create yourself as with Movie Magic Scheduling or Zeni 4, while there are some procedures one must take care to follow, reverse-engineering is completely legal.
Another way to put it, is that among the reasons we have patents, is so that reverse-engineering won't be necessary.