If I remember correctly, you could forget the "inverses" bit and define a group action with only the monoid corresponding to the group.

So not all the "groupiness" is captured by actions.

Also, Bill Thurston's beautiful "On Proof and Progress in Mathematics" [1] contains several ways to understand the derivative, in addition to it being a look into what a very distinguished mathematician thought of the nature of the mathematical pursuit. Pretty similar ideas, explored in depth.

Edit: Apparently this is linked to in TFA :)

[1]: https://arxiv.org/abs/math/9404236