100ms TLS session setup time with RTT <15ms seems rather unacceptable to me.
Can you provide the timings this command produces for you?
curl -s -w 'TCP=%{time_connect} TLS=%{time_appconnect} ALL=%{time_total}\n' https://YOUR_REMOTE_HOST_GOES_HERE -o /dev/null
That should provide an easy to use ruler to compare measurements.