I agree that the triangle-based definition starts breaking down once you get to angles greater than pi/2. I was told in school that "there are no right triangles with such angles, but just imagine that there were" and was then shown some weird (supposedly suggestive) diagrams. I found this unsatisfying and moreover hard to remember.

Of course, you could always just define sin and cos for acute angles only and then extend the definition with trig identities, but that seems rather unmotivated too.