Another idea would be to still run the rest of the E2E tests pre-merge, but as a separate job that only makes itself known if a failure occurs.