Are you sure it's not IdentitiesOnly=yes instead? Or do both work? (I have this line in my config: VisualHostKey=yes )
Configuration options may be separated by whitespace or
optional whitespace and exactly one `='; the latter format
is useful to avoid the need to quote whitespace when
specifying configuration options using the ssh, scp, and
sftp -o option.