It's normally the opposite. Unit tests are too small.

If you unit test all of the objects(Because their all public) then refactor the organisation of those objects then all your tests break. Since you've changed the way objects talk to each other, all your mock assumptions go out the window.

If you define a small public api of just a couple of entry points, which you unit test, you can change the organisation below the public api quite easily without breaking tests.

Where to define those public apis is a matter of skill working out what objects work well together as a cohesive unit.