Why can't the usual comparative advantage arguments work? The USA trades with many African countries, even though there is nothing those African countries can do better than the USA - just like your colonists.
How it works for physical good is clear. But how can one society produce one category of ideas cheaply while another produces a different category of ideas cheaply?
Different brain organization?