Comment by SonOfLilit | Hacker News Reader

Comment by SonOfLilit | Hacker News Reader

Whenever I see a NetLogo based paper, my suspicion is immediately raised. It's not that multiagent simulations are never good models of reality, it's that they are so hard to get right and so easy to shoot yourself in the foot with and not notice. Nonlinear dynamics are hard.

Usually you write a simple model, it does something pathological like diverging to one extreme, you maybe tune the parameters a bit until it does something that fits better with your expectations, then you publish. But the behavior you observed is still pathological, dependent on fine tuning of the model parameters, and has nothing to do with the relationship between the model and reality.

I especially panicked when I saw people on HN arguing we should drop all those SIR-based models for Covid-19 and use multiagent simulations instead.

Also, NetLogo obfuscates from you that usually what you're trying to model is a very simple one-dimensional equation that could be written in three lines of Python, and that you could see the problems with immediately if you'd look at the actual equation.

This is a good example: https://arxiv.org/pdf/1802.07068.pdf

My own review is in Hebrew, but it seems like this is a good one: http://joshuaballoch.github.io/luck-in-life-still-misunderst...

>But the behavior you observed is still pathological, dependent on fine tuning of the model parameters, and has nothing to do with the relationship between the model and reality.

Basically, a mirror of modern DNN approaches: data driven correlations and hyperparemter tuning. Main difference are that with MAS, humans are usually tuning (this is changing though) vs DNN approaches, the tuning is algorithmically driven.

The one benefit I would claim is that with these MAS, you at least know which base assumptions you're really tweaking to converge on states of interest. The question is (as always) if you've accurately: reduced the problem and represented principle interactions in the model. Given the complexity of real world systems MAS try to model, I think this is optimistically incredibly challenging and more realistically, nearly impossible to achieve currently. I think it's still a useful modeling approach that should be explored though I feel it's still in it's infancy compared to numeric, analytic, statistical, etc. models. I think if its taken the surge of computing infrastructure to drive the current successes we're seeing in DNN, it may take an equally larger surge to see improvements in MAS.

Not at all similar in my opinion. Hyperparameters don't usually directly affect the model the DNN implements, they mostly affect the speed of convergence.

Stuff like sensitivity analysis is both easy to do and part of standard operating procedures when working with DNNS, but not when modeling with NetLogo.

Any modeling approach can be implemented/used in an incorrect way. The so-called analytical approaches also often make very specific assumptions on functional forms to be able to obtain analytical solutions. By performing extensive sensitivity analysis one can reduce the drawbacks you mention. Of course, this needs to be done, and you are probably right that many papers don't do this. But that is hardly a problem with the modeling technique, which can be quite effective if used appropriately.

Agent based simulations obfuscate this, both to the reader and experimenter.

Reply on news.ycombinator.com