If you don't have someone in your company scrutinizing everything and spot-checking work that comes across, they will notice and they will take advantage of it. Starting with strong developers to impress a client and then swapping them out with newbies is standard practice.
When we engage with agencies, we make it clear that the people working on our project are expected to be present for the one weekly meeting and that they will be the ones doing the work.
We also reserve the right to "fire" specific agency team members and request a replacement if they're not working out. This is crucial as the agencies will often send teams of three: 1 strong developer, 1 strong communicator, and 1 straggler who needs a lot of supervision from the first two. You pay for all of the developers, so you might as well insist that you're getting what you pay for.