You are also limited by number of access paths into the memory which holds, say, macroblock. I forgot about this issue, sorry.

For block RAM in previous generation FPGAs from Altera there were one read and one write paths. To have two read paths you would need to copy block RAM as many times as you need read paths. This means that if you search for block content inside a macroblock with N parallel accesses you would need N copies of macroblock stored.

Tabula's time shifting tech allowed for up to, I believe, twelve paths into the block RAM, six for read and six for write (they were time-scheduled to 1-read-1-write block RAM operating at six times the frequency). I thought this thing would be a road for superscalar FPGA CPUs, but Tabula was closed.

You can imagine not using RAM at all, but then you will spend other resources in FPGA.

These are problems with video compression I see here. I think they are substantial but not unsurmountable and require a balance to solve. It is just me that I saw the balance is not in favour of FPGA.