No, there's a middle ground where a post is simply de-ranked. Which, absolutely, makes it hard to tell what happened as an outsider. IIRC, basically a "flag" also counts as a bunch of downvotes, in addition to adding up to a threshold where it actually marks a post as "flagged".