A mental model is a model you hold in your mind of how something works, and informs you of its usage, how it can fail, and how to fix it when it does. Mental models are about specific systems and are built over time as you interact with the system; it may not be a perfect representation of the internals but as a model it's good enough to use the thing.
A simple example might be turning the wheel of a car in order to turn the vehicle. How far can you turn it? How fast do you need to turn it to make a given turn? How fast can you be going / how much do you have to slow down to make the turn safely? How about keeping going straight on the road, is holding the wheel still sufficient? When making a turn, do the wheels in the back of the vehicle follow the same tracks as the front wheels?