The RPi's CPUs operate at 1.5 GHz (1500 MHz) or higher, meaning there are 1.5 GHz signals being sent around inside the CPU. The logic in this FPGA under the most ideal theoretical conditions can only operate at around 400 MHz, and for a "real" design, much slower than that - hence my reference to VexRiscv. It's an extremely simple core by modern standards, clock-for-clock it's way slower than the ARM cores in the RPI, yet it's only capable of hitting 200 MHz in (this) FPGA.
To elaborate a bit:
An FPGA has a few major components, but from a logic perspective, the two to focus on here are the "slices" and the routing fabric. The slices of the FPGA implement user logic and are very simple. They contain a few LUTs (look up tables) which implement logic functions. In the case of the Xilinx 7 series architecture, these can be either 6 input 1 output logic functions or 5 input 2 output functions. Other FPGAs can and will be different. By "function", think logic gates. For every combination of inputs, is the output on or off? The logic blocks also contain a number of flip-flops to hold state. There is one for every LUT output, so twice the LUT count (on 7 series). There are a few fixed function components to improve performance of common logic types, such as the carry chain for an adder or multiplexers. Some FPGAs bundle hard-cores for various logic, many have hardware multipliers that can be used, more complex (and expensive) ones can even have CPU cores (Xilinx Zynq and Intel/Altera Cyclone V for instance).
There are thousands of these slices on an FPGA. The smallest member of the Artix family has 2,000 of them, the largest contains over 33,000. The big Kintex and Virtex parts can have hundred of thousands. In order to do anything complicated, you'll need to use many slices to implement logic, which are connected together through the logic fabric. You'll see things like "logic depth" which is the number of LUTs connected in series before connecting to a register. The greater the logic depth the slower the design. The shorter the logic depth, the longer the pipeline. Cue Netburst style concerns. If you don't have to worry about hazards, it's completely fine, if you do, it's a nightmare.