What you're describing is endemic across HN (and tech, tbh). Lots of people on here "know" computers/programming/CS very well. They, naturally, tend to use analogies to computers/programming/CS when trying to explain or "think out loud" in their comments. That's fine. It's what they know. The common problem arises when people forget they're analogizing and begin to see their analogy as ontologically and conceptually identical to the thing they were making an analogy for. This requires a certain amount of ego, echo chambers, and self-valorization, so that they never have to face the actual issues with these analogies.

But as many comments here have pointed out, studying neuroscience, for example, usually makes those analogies seem painfully inadequate. The same is true in philosophy of mind, for example.

Have we hit the limit of the analogy, or have we hit a limit in our understanding? Both neural networks and actual brains have behaviors that emerge from the interactions of smaller components. Neural networks have trivial connections compared to brains, but our understanding of the emergent behaviors seems very limited. To me, this is a sign not that the analogy has reached a point of breaking down, but that our tools aren't sufficient to work on even then trivial connections. I do expect the analogy will break at some point, but I'm not sure we have reached that point yet.

I'm sure that there exist people who get lost in the analogies. Practitioners are generally not confused that ANNs are simplifications of the brain. The questions are which simplifications are most relevant and whether complexities can be added that yield better results. My own research was about reintroducing absolute location. I'm standard ANNs location is relative within a graph model of the network. I'm the real brain blood vessels and other macrostructures deliver materials used to grow and modify the neurons and these affect the network based on physical location. I'm fact, by adding these back in we bypassed the XOR limitation (i.e. Minsky's result leading to back propagation). Concretely, we observed learning XOR over the inputs within a Hopfield network using Hebbian learning modulated by spatially modulated trophic factor).

Good thing there's not a lot of ML being done in Haskell... imagine a Brain Monad tutorial.

Turns out "water in a pipe" is actually surprisingly accurate, just that the "waves" are sloshing around at half the speed of light.

https://youtu.be/2AXv49dDQJw?si=ZxRyiu8WwynUXwf_&t=655

I do not believe that is the case with super "simple" ganglion.