You can. It's just very costly. Rays typically represent just a single sample, so the more probabilistic the projected image becomes (read "unfocused"), the more samples are required to capture it smoothly. This is compounded by the number of lens elements, the material's index of refraction and other complexities. A single element lens adds a minimum of two additional ray/surface evaluations for every sample of which their are many per pixel.

Also, as a separate matter many implementations optimize around the assumption that the projection from three dimensions to two is done in a uniform, undistorted space. For example the kind of projection that can be fully described by a 4x4 matrix. Changing that assumption loses those optimizations, and generally leads to substance abuse and mayhem.

Not an expert, I would guess at least these two things:

1. There are actually around 20 lenses in a camera, not one.

2. A lens converts a ray into a cone, right? And then you need to pass that cone through the other 20 lenses which further modify it's shape, differently at different wavelengths.

I would guess this is extremely computationally expensive done naively.

You can at the cost of some light bounces.

Most modern render engines render rays from the camera into the scene. This speeds up rendering a lot but also means rendering caustics will need a huge amount of samples because the probability of seeing light from a camera through glass is very low. Bi-directional path tracing fixes some of these problems.

So it can be done but always at a cost (rendering times).

In Blender I have done this a couple of times and even used different IORs for the RGB colors to get real world results.