My experience is that if there is a VPS located in the U.S., which only has a static page/file, and there is no CDN, whether it is access from Japan, Hong Kong, Singapore, or from mainland China, the speed is similar. Access from mainland China will not be significantly slower. I did this test over and over again.
The reason your site 'slows down' may be that your site is blocked, or the embedded Javascript (such as Google Analytics, Adsense, DISQUS) or CSS/Web Font (such as Google fonts) has listed on GFW's blacklist. I feel sorry that my last reply may be not made it clear, I mean that the point of this project is not all the truth (why your site is slow), or even a little misleading. Since running website for several years, I must be honest point out that if the mainland China market make sense to you and you want to keep your website always smooth, the only way is to place the server in mainland China or Japan, all other testing, optimization is a waste of time and money.