Definitely will be modifying the experience to be fully handsfree.
I think the context issue is what can make this actually work or not. Currently it's not built out but my thinking is to have a short context of what was commented on i.e. 10 seconds before the comment. That way you can jump back into the conversation from a new comment left.
I think the context can also be determined by how long it's been since you listened to the last audio - meaning a comment left after a week might have 60s of context vs a comment after 10 mins might just have 5s.
And yes, the UI isn't great at showing what you already listened to now but that needs to be obvious too.