This works because casts are allowed to quietly create values whose types are wrong. It would have been better if the cast added a runtime check, or at least we distinguish sound (checked) and unsound casts the way C++ does.
I think Haskell avoid this by actually requiring you to write sound conversion functions between phantom types (it helps that phantom types don't involve any visible state at runtime).