While I find the question puzzling, i find the description lacking. The whole explanation of anon maps is redundant, since the behavior of anon maps seems to be the same as the behavior of plain memory - its evicted on low memory conditions with LRU behavior and there is no anon maps specifics mentioned.

To recap: swap makes anon memory pages reclaimable, by offloading it to disk. This does not explain anything. Its basically "swap allows anon maps to be swapped"; so what?

>> You need to opportunistically handle the situation yourself before ever thinking about the OOM killer.

Or, just rely on OOM killer, minimal swap(so the application dont agony in low memory conditions) and restart policies. There are no reason to do the kernel work handling memory over-commit.