The issue in this case was Opus was pretty crap at C. It kept introducing segfaults.