Does this always happen or only sometimes? In Google calendar, if the event is modified through the API, there is an option to suppress change notifications, and this can prevent non-google calendars from getting invites.

Also did you create a Google account using your fastmail address? I haven't tested it, but it's possible this causes a hairpin route where Google supresses the outbound email since it thinks it can directly modify the calendar.