You might want to read Hamilton’s defence of Elgin to understand how they saw the difference between legitimate and illegitimate acquisition of antiquities.
Furthermore theft of antiquities isn't quite the same as slavery, genocide, apartheid... Although culturally relevant, it does form part of ethnic cleansing - which the British Empire excelled at.