Or do I have that backwards...
The far away observer sees the falling one infinitesimally approach the event horizon, but never cross.
The more sensitive the instrument, the longer you can observe, and this doesn't really have a limit - as sensitivity approaches infinity, so does the length of time you could continue to observe the object.
But for practical purposes, we would not see a black hole as some sort of weird psuedo-magnet with all sorts of junk stuck to the edge of the event horizon.
For example, lets say you dropped a beacon that flashes every second into a black hole. As it approached the event horizon, you'd see the flashes only happen every 2 seconds, 4 seconds, minute, hour, decade, etc. Meanwhile, the length of those flashes are getting longer at the same rate, while producing the same number of photons, so the light gets dimmer and dimmer.
I guess you're assuming zero velocity by that person. But if there is velocity, wouldn't the fall not be straight towards but in an ever shrinking/decaying orbit trajectory?
I should have remembered the film Interstellar, where an observer on a spacecraft orbiting a massive planet (itself orbiting a black hole) feels decades pass whereas the landing party to the massive planet experience about an hour go by.
Even wording this sentence is difficult! I'm reminded of the Persian word for time, which literally means "the thing that passes".
If you were falling into a wormhole, you would die of old age before getting there while outside observers would see you fall in quickly.