Unit tests are deeply coupled to the internal structure of your system - refactoring often implies changing unit tests, which opens the door to bugs where the code and test both change to match each other.
As you say, integration tests validate your public API, which from a 'correctness' point of view is really the only thing you care about, not the internal structure of the system. That's why I love integration tests, you can make sweeping refractors without needing to change the tests, because the test will still tell you whether the behaviour of the whole system is correct.