Another aspect is software optimisation. For example, last time I looked at ffmpeg there's tons more hand-crafted x86 assembly than hand-crafted ARM assembly (been a while, not sure what the current situation is).

In general, ARM optimisation has caught up quite a bit, although still lags behind in some places. RISC-V still has some ways to do. For example, for Go:

  [/usr/lib/go]% (for f in **/*.s; print ${${(s:_:)f:t}[-1]}) | sort | uniq -c | sort -hr | head -n10
     86 amd64.s
     72 arm64.s
     63 s390x.s
     56 s
     48 arm.s
     47 386.s
     35 riscv64.s
     32 ppc64x.s
     22 loong64.s
     20 mips64x.s

  [/usr/lib/go]% wc -l **/*amd64.s | tail -n1
   28801 total

  [/usr/lib/go]% wc -l **/*arm64.s | tail -n1
   21956 total

  [/usr/lib/go]% wc -l **/*riscv64.s | tail -n1
    7804 total

Rough measurement of course, but at least some code paths for your RISC-V will be slower just because it's not optimised (yet).