It was only possible in the Victorian period for the wealthy elite. If you were born into a family that owned vast swaths of land or used to run slave plantations in the Caribbean you were set for life and could devote your life to less crude pursuits than the gain of capital. You would also have a ready built network of others with similar wealth and social expectations.