Git, like most versioning systems works by adding a file to a repository. Then subsequent changes are added to the repository by saving only what has changed in the file.

In the case of a bitmap image think of a grid made up of pixels. Each pixel is stored on the file as binary data (0's and 1's). Anything that affects that pixel will change it's value in the file.

With a vector file there aren't pixels necessarily. The drawing is saved as shapes and those shapes have properties. These shapes and their properties are actually saved as text (sort of like HTML markup).

So as the file changes the changes in those various file types are more drastic. With a bitmap there will be greater changes so overall more space is used to store the file and it's history. Also, because it's binary it's difficult to see the changes in the actual file (without opening it in a photo editing program). With a vector file only small amounts of text would be changing and you can literally see the changes in the file and make sense of it.

Hope that helps.