Sometimes you can take a pretty good guess who it is based on the topic and the references (e.g., citations to in-press papers on obscure topics is usually a give away). Other times, though, it's difficult to tell who the authors are.
People often also think they know who the author and/or reviewers are when they are really totally wrong. I've reviewed papers and seen other reviewers cite my papers in a way that I thought "damn, that looks like something I'd write" when it was not. Similarly, I've seen reviewers think they know who the author is when they're completely off.
I personally think anonymous reviews are critical, because it keeps some modicum of shielding of reviewers against retribution for negative comments. It's not 100% anonymous of course, but it's anonymous enough to create enough doubt about who the reviewers and authors actually are.