As others have commented, you would indeed say, “I have an appointment with my GP.” However, people would certainly know what you meant if you said “I’m going to see the doctor.”