This was the message after the appeal:
"We still found the app is inappropriately using background audio because it is recording audio in the background for an indefinite period of time, which is not appropriate."
On the other hand I assume that you don't need to keep long audio samples to run your alarm-recognition algorithm and just discard the audio almost right away.
That's just a wild guess of course, it would just seem reasonable to me if the Apple reviewers saw it that way.
I also found other people with the app being rejected for the same reason after asking about it in the apple dev forum.