This was well reported for major digs like WTC/WFC, which created Northwest Battery Park City.
It's also seen in local landmarks like the Ear Inn on West Spring (one of the oldest bars in the city) which has an old painting inside showing the shoreline being not far outside the building, whereas it is currently a full block or two
For more: https://en.wikipedia.org/wiki/Lower_Manhattan_expansion
The ships returning from England would load up with rubble to use as ballast.
Yup, all along the East Side. This is why you’ll find English weeds growing on Manhattan.
Edit to include text from the plaque commemorating the basin: BENEATH THIS EAST RIVER DRIVE OF THE CITY OF NEW YORK LIE STONES, BRICKS AND RUBBLE FROM THE BOMBED CITY OF BRISTOL IN ENGLAND... BROUGHT HERE IN BALLAST FROM OVERSEAS, THESE FRAGMENTS THAT WERE ONCE HOMES SHALL TESTIFY WHILE MEN LOVE FREEDOM TO THE RESOLUTION AND FORTITUDE OF THE PEOPLE OF BRITAIN. THE SAW THEIR HOME STRUCK DOWN WITHOUT WARNING.... IT WAS NOT THEIR WALLS BUT THEIR VALOR THAT KEPT THEM FREE ......
And broad-based under all Is planted England's oaken-hearted mood, As rich in fortitude As e'er went worldward from the island-wall.
ERECTED BY THE ENGLISH-SPEAKING UNION OF THE UNITED STATES * 1942 *