Neural networks are actually somewhere in between. They don't directly operate on symbolic expressions or explicit logical rules. And while they rely on probabilistic aspects for training (and sometimes for inference), they rely more on continuous-valued transformations in extremely high dimensional spaces. But if your goal is human-like intelligence, they are a pretty good bet, because we know the human brain also doesn't perform symbolic reasoning at its core and these things only emerge as high-level behaviour from a sufficiently complex system. But it also makes neural networks (and us too) prone to failure modes that you would not see in strictly symbolic reasoning processes.
Yes, this seems to be what the symbolists always forget. We don’t use symbolism like this, we just have very dense neural connections that emerge from scale and approximate it
That's not really what symbolists actually argue. Read Fodor and Pylylshyn 1988 to better understand how symbolists view the relationship between symbolic representation and connectionist models. What you're saying is akin to saying there's no point in doing neuroscience or trying to understand neural-networks as its just particle physics deep down.
So if inventing the airplane how long should one stick with a flapping bird wing design?
That is not a good analogy. Symbolism has given us lots of useful things, including SAT/SMT and theorem provers.
The point is that while biology has been a great source of inspiration toward technical advances it’s only a guide, it can’t guarantee there isn’t a better way.