Tay was presumably the English version of XiaoIce, which apparently has its own paper[1]. It states that they combined a Markov decision process to select between conversation modes, a retrieval-based response generator for curated answers and a GRU-RNN[2] response generator for free-form ones.
[1]: https://arxiv.org/pdf/1812.08989.pdf
[2]: https://en.wikipedia.org/wiki/Gated_recurrent_unit