But nothing prevents such a system from working with arbitrary files, as long as there is a tool to create a "patch" or a "diff" (that is, a description of a change) and a tool that can apply such "diff". Git does have a mechanism to plug in both of these tools, depending on the file type.
I have a very vague idea of what CAD files are internally, but I suppose they are some kind of a database, and they are not colossal (unlike e.g. video files or such). They certainly could use this mechanism to allow tools like git (or hg, or maybe even perforce) be used in workflows involving CAD files, with human-understandable and reviewable changes.
I wonder if someone has ever tried that. I can't be the first person to have this obvious idea.