This type of computer is also called a "parametron" (https://en.wikipedia.org/wiki/Parametron). A Japanese researcher named Eiichi Goto independently invented it around the same time as von Neumann, and developed it much further than von Neumann did, so the idea is more often associated with Goto than with von Neumann.
The basic idea is that if a nonlinear harmonic oscillator is driven at twice its resonant frequency, it will oscillate stably in either of two phases. The two phases represent "0" and "1". If the driving signal is switched off and on again, the oscillator will arbitrarily "pick" a phase to stabilize on. If it's exposed to an input signal from another oscillator as it's turning on, it will always pick the same phase as the input signal. This makes it possible to copy a "0" or "1" from one oscillator to another. If the oscillator is exposed to input signals from several other oscillators, it will pick by "majority vote". Finally, a NOT-gate can be built by inverting the signal polarity. This set of primitives is sufficient to build arbitrary logic gates and flip-flops.
Goto's paper has an excellent explanation and more detail:
Goto, E. (1959). The Parametron, a Digital Computing Element Which Utilizes Parametric Oscillation. Proceedings of the IRE, 47(8), 1304–1316. doi:10.1109/jrproc.1959.287195
(PDF available at https://sci-hub.tw/10.1109/JRPROC.1959.287195)