The assumption is, that a patch is perfectly diffuse and it reflects in all directions equally. Imagine a fish-eye-lens perspective (actually a hemisphere) from a patch into the world. The projected size and visibility of all the other patches gives you the proportion of energy transferred to/from that other patch.
> Was it a performance matter ...
Computing the final color of a patch is somewhat computationally expensive. It involves either inverting a giant matrix, or iterating until you reach a stable result. The end result is not a rendered image, it's a perspective independent light distribution in your scene. You still need to combine that with another rendering technique, like ray-tracing or rasterization to actually get a picture.
Computer games of the era you mention (e.g. the Half-Life series and other GoldSrc/Source engine based games), did radiosity in an offline, pre-computation step and stored the patch colors in a texture (this is usually called a light map). Using the light map at runtime is pretty cheap.
> ... or was it a physical model limitation (you can't simulate all light phenomenons) ?
As I described above, radiosity assumes perfectly diffuse reflectors. It can't do mirror-like specular reflection of light. Ray-tracing and radiosity actually complement each other really nicely in that aspect, as mirror like reflections are trivial to do with ray-tracing and you can integrate light maps for indirect, diffuse bounces.
That said, with modern hardware, doing radiosity in real-time even for reasonably sized scenes has become feasible. There is e.g. the proprietary "Enlighten" engine that claims to do real-time radiosity. There are other approaches that have become feasible as well, like doing real-time photon mapping. Ray tracing hardware finally being a thing is also certainly a game changer. Well over a decade ago, Crytek did some work on voxel grid diffusion based stuff and Nvidia worked on integrating cone-tracing into that. I have little idea tough what modern game rendering pipelines look like, I've been pretty much out of touch with that industry for quite a while now.
I don't think lightmapping went away. I still see it even in AAA titles today. Performance-wise, for a static scene, baked lighting is tough to beat.
The actual specific radiosity algorithm (diffuse only, form factors and matrix solving etc) proved to be too unwieldy, requiring huge amounts of effort and care to get good results. This is in contrast to Monte Carlo path tracing, which doesn't require extreme care with meshing, doesn't assume diffuse BRDFs and is really well suited to modern massively parallel architectures. Consequently it completely took over production rendering, and game rendering is next.
To get good results on a complex scene you need to increase the subdivision count of the radiosity calculation by a lot, which will be slow unless you're smart about it. And radiosity only accounts for diffuse bounces, leaving out glossy and specular reflections.
The Wikipedia page has a section on Limitations.