Interesting!
Relevant work:
1. What's one of the most important hyperparameters? Architecture: https://arxiv.org/abs/1806.09055
2. Dropout: https://papers.nips.cc/paper/5032-adaptive-dropout-for-train...
3. Learning activation thresholds: https://datascience.stackexchange.com/questions/18583/what-i...