The same thing was true of H264 when it first came out. Reference encoders/decoders are always incredibly slow -- they're written to be correct, not fast.
AOM dev here. I'd love you to be right, but that's unfortunately not the case. They're written to improve VQ.
Regarding the "written to be correct" part of your sentence:
There's no emphasis on correctness. For monthes, aomenc has been accidentaly using different transforms sets for 8 bit and 10 bits. It also used to compute its RDO based on the legacy probability model (non-CDF) although the result was binarized using the new one (CDF)). Moreover, serious regressions (decoder can't decode the encoder's output, lossy lossless) are introduced almost every day (!), fixed the next day.
Now, regarding the "not fast" part of your sentence:
There's been a lot of work on SIMD (Current aomenc performance is measured with SIMD optimizations). Actually, the code features complex mechanisms (C header file generation from Perl scripts at configure time) to handle the dispatching between the optimizations. To make the matter worse, almost all the pixel prediction code is physically duplicated in two versions: the generic code that uses 16-bit pixels, and the 8-bit optimized code that uses 8-bit pixels. All of these are actually huge hindrances to performance-unrelated developments.
It really wouldn't be that surprising if we're still using video encoders with efficiencies somewhere in the h264-AOM range 1000 years from now. Cases where the tradeoffs make higher efficiency worthwhile are hard to come by.
H264 Reference Encoder wasn't even tuned for anything. Not even quality. Hence why we not only get much better speed in x264, but also higher quality.
And the Reference Encoder ( JM ) was only 250 times slower when it first came out. It truly was a pieces written to be correct rather then anything else. And there was some easy wins to get it 10x - 20x faster within a matter of weeks.
Now Av1 is nearly 10000x slower, that is if you manage to speed it up 100x it is still 100x slower. The codec right now was tuned for quality metrics. They were trying to hit PSNR, SSIM and even VRMF at 30% higher then VP9 at all cost. Which is not the same as JM.
I dont doubt they could speed it up. But by how much. Some even mention it is only usable by companies with Scale, well this is so far from truth. You dont gain much economy of scale once you past certain level of compute. What cost $1M dollar to encode at facebook now cost $1B or $10B as it stands. People seems to have a real lack of sense in scale of how much 1000x is. Or even 10K x. Your compute cost is going to be dictated by CPU manufacturer or silicon prices. And that 30% of bitrate reduction is not going to save you money from the insane increase of compute. Especially if you are Netflix with Appliance sitting at ISP's DC.
Not long ago I asked how are we going to get the encoding speed to some sane's time. Like 10x of x264 High Quality. Turns out it will very likely have to disable certain tools set now being used, which means a reduction of quality. But I believe it will still be better then x265 at the end. The tools available for AV1 is quite massive.
Right now we have xvc which is looking more like an H.266 encoder, and offer 30% improvement over AV1 if you want to believe PSNR and SSIM. But it is even slower then AV1.
Even without rotation, MV search can get sophisticated and not exactly intuitive in a typical encoder. And with HEVC having "merge" that win that much bits, it's not like there was so much bits left to earn with the predictor itself.