Might be silly question but is there a technique to rapidly program FPGA's without interrupting other processes. Say I have multiple soft-CPU's and only want to use my gates to enable ethernet once its needed or the user plugs in?
There are, however, several limitations to it. Clock cannot be changed, for example, and usually neither can I/O, specially high-speed transceivers. This has been improving (Ultrascale Xilinx allows for reconfiguring I/O), but you still have to reserve area for reconfiguring (meaning literal area, as in a geographic region in the FPGA).
However, I/O versatility as you suggested has very few advantages to it. You need the reserved logic for ethernet to be programmed at when you plugin. Why would you leve it unprogrammed? If is simply disabled, it won't use any extra power resources, and your soft-CPU's won't be able to take advantage of these resources while you are working. Maybe you could use the area for new soft-cpus, but then you'll hit the problem of over segmenting your design and allowing for less optimization. This would inevitably impact timing constraints and area usage.
Also, FPGA programming may take minutes to finish, and always at least a few seconds. This will be very noticeable by an user and not very efficient if it has to be done frequently.
There are, of courses, good uses for that. But there is also a lot of effort on doing it right and you always risk overdoing it.
Also, for some designs you can mitigate the reconfiguration time issue by having two regions and draining requests to one of them, before doing an update. Most of the Xilinx tooling for OpenCL does this kind of thing by default (4-6 "opencl kernel" regions.) But of course it's not always an option to give up that much space...