Interesting! I wasn't aware that anyone was doing double-anonymous peer review. I would love to know how that is working out.
One of the biggest challenges was to decide what to do about ArXiv and pre-prints in general - most researchers keep track of new papers on their area, and therefore it is likely that they will be assigned as reviewers for a paper that they have already seen (and for which they already know who the authors are). This issue came up for a vote a couple years ago, and the chosen answer was to not disqualify papers for this outright but to kindly ask authors to refer from doing so.
At the same time, "88% consider double-blind reviewing at ACL conferences to be important", and only 14% of reviewers would penalize a paper for not including a relevant pre-print. This seems to me like a fair compromise. There is also work in progress to either talk to ArXiv to allow for author masking, and/or to create a new pre-print server that allows it. There is a very detailed report on the results of the survey in [1].
A second issue is whether the reviews should be made public, because it could compromise the double-blind process. The general result of the survey [2] was against it, where "support for public review tended to be inversely correlated with reviewing experience (and) female respondents were less likely to support public review than male respondents".
[1] https://www.aclweb.org/portal/sites/default/files/SurveyRepo...
[2] http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019...
https://blog.siggraph.org/2018/10/the-magic-of-technical-pap...