This is really exotic space. Time dialation immediately outside of the event horizon would make any matter fall so slow that it would take millions of years of time for an observer to see that material cross the horizon.
Growth-wise, you have to keep in mind that the black hole doesn't grow when matter crosses the horizon, but when the matter gets close enough to the horizon that mass of black hole + mass of new matter creates a blackhole of a larger radius that encompasses that matter.
See especially the section on entropy.
https://en.wikipedia.org/wiki/Maxwell%27s_demon
That is an example of one way in which information can be exchanged for work.
Specifically, the entropy within the box gets converted into information inside the demon's "head", which eventually gets radiated away as entropy outside the box. This way, the box's entropy falls, but the entropy of universe as a whole is raised (or at least maintained).
If information can be destroyed, physics are no longer reversible. Playing things in reverse would give you events that occur for no reason. This is hugely important not just for particle physics, but also thermodynamics.
Physics is a deterministic theory, even at a quantum level where we call determinism unitarity. (Leaving the measurement problem & collapse of the wave function aside for a moment.)
Determinism means, we can start with one state of a given physical system and basically hit the "fast forward" or "backward" button to see what the state was like in the past and what other state it's going to evolve to in the future. This assignment of future states to past or current states and vice versa is unique and one-to-one, hence deterministic:
past -> future
A --------> X
B --------> Y
where the arrows indicate evolution in time.
This is what allows us to make predictions for the future outcome of experiments or draw conclusions from experimental results in the first place.
Now, the problem with black holes and in particular their evaporation through Hawking radiation is that Hawking radiation does not in any known way depend on what once fell into the black hole but, instead, just on the black hole's total mass. So let's say you start with two different mass distributions A, B of the same mass M and each collapses and forms a black hole BH of mass M. You could argue that the original matter is still somewhere inside the black hole, so the black hole's internal state still depends on the past matter distribution and we should indicate this by talking about black hole BH_A and black hole BH_B. So we have:
A -> BH_A
B -> BH_B
Then Hawking radiation sets in and both black holes vanish eventually and give way to a state of Hawking radition H:
A -> BH_A -> H
B -> BH_B -> H
As mentioned before, the Hawking state H really only depends on the mass M of the black hole, so since both BH_A and BH_B had the same mass, we will also get the same final Hawking state H. As you can see, the mapping {past events -> future events} is no longer one-to-one: From H alone you cannot draw any conclusions about the original state anymore. (Was it A or B?)
This is (one half of) what is meant when people say that information is lost (about the original state).
The other half is that a classical (eternal) black hole state is a so-called pure state from the point of quantum mechanics: There is never any doubt about its precise state. It is always perfectly described by only its mass (and angular momentum and charge)[1]. Hawking radiation, however, is not such a state once the black hole has fully evaporated. Instead, at that point it is a so-called mixed quantum state meaning that it could be any of a whole "mix" of states. We can only give a probability distribution for what the state is in reality; in Hawking's case it's a perfectly thermal distribution. (Side remark: I should point out that such a mixed state is not the same thing as a superposition in quantum mechanics! In particular, the probability distribution here is not related to any wave function.)
Anyway, what this means is that we have now evolved from a black hole state BH, that was precisely known, to another state about which nothing can be said anymore outside of the probability distribution:
BH -> Hawking_1 or Hawking_2 or Hawking_3 or …?
This constitutes a loss of predictability.
TL;DR To sum up, there are two aspects: The loss of information and the loss of predictability. Both are just the two sides of the same medal: The breakdown of determinism.
This is also the principle that led to our theoretical understanding that the maximum amount of information that can be contained in a volume is bounded by the _surface area_ of the volume. This is the Holopgraphic Principle: https://en.wikipedia.org/wiki/Holographic_principle.
There's a great PBS Space Time youtube video on this concept, which explains it far better and more accurately that I ever could: https://www.youtube.com/watch?v=9XkHBmE-N34
If you are travelling with a probe towards the horizon though, you would be able to calculate your escape velocity, so you could tell if you are inside our outside the horizon.
Its kind of awkward if you think about it... its seems like causality has 2 versions after the horizon.
[0]: https://en.wikipedia.org/wiki/Black_hole_information_paradox
This is merely conjectured by some physicists. In reality, we have no clue what happens.
The question necessarily treats space and time as two seprate entities, which generally works fine. Relativity starts to break that question. Different inertial/non-inertial observers will get different answers. Not a huge problem for physics, because we can easily figure out the answer for each observer. But if it was super important to you what event happened first, you won't get a satisfying answer, unless the two events occur inside each other's light cones. Then there's no ambiguity.
So we get to black holes, they seriously break this question, the light cones are completely messed up.