I am currently using the open source tomato firmware. However, since there is a bug/feature in the router so that I cannot flash an image too large, or otherwise it would not work. Also, the configuration is limited to 32 KB, if configure too much, then the configuration file will become gibberish and some random feature in the router would be missing, and required a factory reset to fix. So, I am stuck with an older version of tomato which guarantee some kind of vulnerability is not fixed.
Not sure what I can get in the form size of a router. Raspberry pi may work but too few ports available. I heard that the CPU would get hot for intense network traffic.