If there's an architectural design up front then there's a need to source materials that fit the plan, as you describe.
The opposite approach was taken by a friend of mine who was, for a decade, a prolific builder here in Australia .. he continuously had two or three houses on the go in staggered completion (clean builds and|or significant rennovation) that all sold well for their design, uniqueness and craftsmanship.
His starting point was salvage yards, looking for cheap big statement pieces; bay windows, big wall cabinets, doors, window sets, impressive looking ketchen sets, big counter tops, etc.
The next step would be to design a house plan that fitted around quality salvage and well suited the site for views, access, etc.
His arc in life was art school in Victoria followed by | interleaved with a salvage job that dismantled entire (small) towns ahead of dams, flooding, other projects - they prepped wooden houses for moving elsewhere, disassembled structures to "flats", reclaimed historic bridges, etc.
Then came the building that built up his cash reserves, then a big rural property with sheds, glassblowing studios, metal work, etc.