What ICANN aren't admitting is that while there was no formal RFC standard to the output of WHOIS ICANN themselves had mandated the output format of the WHOIS output. To go live with your shiny new gTLD you had to pass a test of your whois server output. It was strict enough that we had to remove an extra blank line is the disclaimer at the end of the output.
Every whois server I ever checked while writing my org's version supported UTF8.
The data has to be publicly available so authentication and encryption doesn't matter. In any case there is currently nothing requiring RDAP servers to authenticate anything.
The amount of search ability I've seen in most rdap servers so far is pretty limited.
The RDAP output is json but its complicated and there are so many options that realistically everyone is developing to pass the icann profile so no different to whois, and in the end we still live in a post GDRP world so all the useful data is redacted so none of this matters anyway.