But for it to destabilize the orbit of planets through the orbit of a binary system, it is quite plausible once the blac hole has gained sufficient mass for it to count as a binary system and not simply a planetary mass black hole orbiting a stellar mass sun[1].
[1] https://arstechnica.com/science/2013/01/binary-star-systems-...
First the outer layers of the sun are not very dense, but as a larger fraction of the matter for the black hole comes from the sun the slower (relative to the sun) the black hole would get. The slower the black hole is that closer it would come to the center of the sun. The closer to the center of the sun the denser the sun is.
The pressure of the sun is at least 10,000 times greater than the center of the earth which is 3,500 kilobar. Wouldn't the amazing gravity gradient and the 3,500 kilobar pressure result in a very well fed black hole that would double in mass within say a few days? Sure a accretion disk would form and start pushing back the matter at the north and south poles to reduce the feeding rate.
Sure black holes generally increase is size slowly, but they aren't usually inside a gas cloud of 1.4grams/cm^3 at a pressure of 3,500 kilobar and having an entire suns worth of mass to provide resistance to the accretion disk allowing for matter to fall in quicker.
I've heard numbers like you mentioned for atom sized size black holes that fell within the earth, but the main problem is that the likelyhood of swallowing an atom is so small that it grows incredibly slowly.