I'm not sure I can give a definitive answer, but there's a few difficulties in making it expand. One is that in the kernel, you can only reliably make large contiguous kmalloc() allocations at init time; another is that the dentry cache is highly optimised for parallel access (most lookups will proceed without taking locks under Read-Copy-Update).

In most cases memory pressure will tend to naturally limit the dentry cache size - the "perfect storm" here was almost zero memory pressure combined with a process doing a lot of negative lookups on an essentially endless list of unique filenames. For such an unusual situation, it's probably reasonable to ask the administrator to manually tune things, rather than building a more complex runtime-resizing hashtable that almost everyone won't need - especially since the failure mode is a graceful performance degradation.