Arbitrary semigroups clearly don't describe the real world. Nor does this prevent us using particular groups to describe particular quantum fields.