Search for "there was no 221 Baker Street during the years of the Sherlock Holmes stories".
Some authors can't always go to this level of detail. For example J. K. Rowling was in Scotland when writing the first Harry Potter book, as a result she describes Kings Cross from memory, confusing it with another London Station, so her description of it does match the real world location.
The article is all about the stuff in Holmes' rooms.
(that they agree with the current museum building is possibly due to the current building being chosen because it fitted the description - rather than it being known as the correct address)
edit: Apropos Conan Doyle:
A cabbie picked up the great writer at a Paris railroad station to drive him to his hotel.
Upon parting the cabbie said: "Goodbye mr. Doyle".
Conan Doyle taken aback by this responded: "How do you know who I am, do you recognize me from my picture?"
the cabbie: "No, sir, I've never seen a picture of you before, but I read you were going to be vacationing in France, your coat has the cut of an English gentleman, you have an ink spot on your finger indicating you are a writer and the train I picked you up on hails from the coast. All these clues together made it very likely that you were the great writer himself."
Doyle: "Amazing feats of deduction, you are worthy of being an incarnation of Sherlock himself!"
Cabbie: "Oh, and one more thing, your bag has your name written on it".
There's a museum/gift shop there now.
The pictures that I can find now online look a lot more residential than what I remember from 2000 or so. I remember that it was a granite-slab front office building with a pla
The location of the original 221 is not known for certain because of street renumbering and destruction during the Blitz.