Lovely sentiment but it doesn’t translate to actual immigration policy.
And, chances are, these two presidents were probably talking about white immigrants, given the racism and slavery of the time. Are you familiar with James Madison’s policies with Native Americans? He privately thought they couldn’t be civilized and wanted white immigrants to eventually mix with and displace them into irrelevance.