Just be honest and tell him it's not working out for you.