Convolutional networks easily get up there, especially if you add a third dimension that the network can travel across (either space in 3D covnets for medical scans, or time for videos in some experimental archetecture). Say you want to look at a heart in a 3D covnet, that could easily be 512x512x512 for the input alone.
In fully connected models, for training efficiency, many features are implemented as one-hot encoded parameters, which turns a single caragory like "state" into 50 parameters. I think there is some active research into sparse representations of this with the same efficiency but I've never seen those techniques, just people piling on more parameters.
A further point is that even if the model has relatively few parameters, there are advantages to having more memory--- namely, you can do inference on larger batch sizes in one go.
- No control flow instructions (though apparently some operations can have a repeat count)
- Fundamentally simple architecture
This allows them to get through validation and tapeout very quickly.
The SOTA networks are around 300MB+...
Since it appears you're in the deep learning hardware business, what would be the impediment to using eDRAM or similar? eDRAM is too costly at those sizes for general purpose processors, but I imagine the reduced latency and increased bandwidth would be a huge win for a ridiculously parallel deep learning processor, and would definitely be a tradeoff worth making.
Okay, so about eDRAM. There are two types of eDRAM: On-die and on-package. On-die eDRAM refers to manufacturing of DRAM cells on the logic die, which would be a big boon in terms of density since eDRAM cells can be almost 3x as dense as SRAM. The problem however, is that on-chip eDRAM has been impossible to scale beyond 40nm, which mitigates any advantages you would receive from using eDRAM.
On-package eDRAM is more interesting but the primary cost in memory access is the physical transportation of the data, which is a physical limit and can't be circumvented. You can call it all sorts of fancy names such as "eDRAM", but the fact of the matter is that you're still moving data. For reference, the projected cost of movement of a 64-bit word on 10nm (ON CHIP) according to Lawrence Livermore national laboratories is ~1pJ, whereas the cost of a 64-bit FLOP is estimated to be 1pJ also. As you can see, the cost of data movement dwarfs the cost of computation.
Of course, you gain a lot compared to DRAM, but HBM can offer the same efficiency gains of course.
Didn't meant to be rude with the first response. Let me know if you have any other questions, I'd be happy to answer them :)