I implemented publicly-verifiable macaroons as a PoC and found them reasonably ergonomic: https://github.com/ecordell/watchstander

(docs are sparse, I wrote an accompanying doc that might help: https://docs.google.com/document/d/1AU9bwpMYlnWBlwSIiwNyse0N...)

The basic idea is what you described: append only asymmetrically verifiable signatures.

As with most things Macaroons, the harder part is developing a caveat language and verifiers that actually meet your needs. And convincing people that they're a good idea.