I read it much the same way, as a rather insulting thing to say, especially since you didn't go into what was said afterwards and you didn't preface the kind of relationship you had. It's not on the reader to understand what you meant, it's on the writer to write what they mean in a way that the reader understands the intent.