You might want to look into OSMCoastline, a separate piece of software written specifically to make the coastline usable for renders: https://osmcode.org/osmcoastline/