From Oscilloscope to Wireshark: A UDP Story
mattkeeter.com
mattkeeter.com
Some years back, Wes Irish at Xerox PARC tracked down one of the great mysteries of early coax Ethernet - why was throughput so much lower than theory predicted? For this, he got both ends of a building-length coax with many machines on it connected to one office, so he could plug both ends into a storage scope. If the waveforms disagreed, somebody was transmitting when they shouldn't. Storage scopes with large storage were rare then. It was an expensive LeCroy unit.
After the end of each Ethernet packet on coax, there is a brief "quiet time", and then the next packet can be sent, beginning with a sync pattern. The hardware detects if what it is sending does not match what it is receiving, which indicates a collision. Both senders stop, wait a random time so they don't collide again, and retry. This is how "carrier sense multiple access - collision detection", or CSMA-CD, works at the hardware level.
This setup revealed that something on the cable was transmitting a single spike after the end of each packet, during the "quiet time". That reset the "quiet time" timer in the network interface, which inhibited the transition to "look for sync" mode. So the next packet would be ignored. The quiet time timer was at a very low level - software did not see this event.
What came out of looking at the waveforms was the surprising result that the spike during the quiet time was not coming from either the data source or the destination, but from something elsewhere on the cable. The spike was not synchronized to the packet just sent. With the waveforms for both ends of the cable visible, speed of light lag revealed both that this was happening and where it was coming from, as distance along the cable.
It turned out that several brands of network interface used a part which contained the quiet time timer, the sync recognizer, and the transmitter power controller. When the timer ran out, the device did a state machine transition, and during that transition, for a nanosecond or so, the transmitter turned on. It wasn't supposed to do that. This generated a spike on the cable, resetting every quiet time timer and causing the next packet to be silently ignored by all stations.
The network interface didn't need to be active to do this. Being powered on was sufficient. One device with that part could halve the data rate on a coax Ethernet. Thousands of network interfaces had to be scrapped to fix this.
Also, some things never change: the scope that Matt was using for this is (still) an expensive LeCroy unit...
There was quiet faxing of waveform pictures to certain IC and board manufacturers. Not much publicity. This was in the 1980s. Ethernet was a niche product.
E.g. I had a student once who was frantically trying everything on the raspberry pi to fix some motor control script he made. I suggested he should just write a minimum workable code first and then check if there is a signal on the wire. Turns out the wire was not connected.
Especially with hairy problems looking at the actual signals can be useful. Granted, doing so for UDP/Ethernet frames might be overkill, but it is nice to see it can be done.
This article is focused on the QSGMII PHY chip, without which it would be practically impossible to use an oscilloscope for any meaningful troubleshooting.
1000Base-T is the most commonly used Ethernet variant in use today. It uses four balanced differential pairs for signalling. Each pair transmits and receives simultaneously. This works because 1000Base-T is a point-to-point connection and the PHY contains a hybrid that subtracts the voltage being transmitted from what is received. This makes the problem of "tapping" the interface harder because the third party observer sees a jumble of data and cannot easily determine what data is coming from either of the two connected devices. Each pair can send/receive five possible symbols, represented by voltages of +3.5, +2, 0, -2, -3.5 volts. In addition, the PHY "scrambles" the data being sent with a LFSR (https://en.wikipedia.org/wiki/Linear-feedback_shift_register) seeded with either a "master" or "slave" value. (The master/slave identity is negotiated between the two peers when they detect each other.) It gets even more complex with hamming codes and an abstraction layer to separate the control/data planes. There's also a complicated DSP for line equalization.
BTW, I do own one of these: https://www.amazon.com/midBit-Technologies-LLC-SharkTapUSB-1... which can make the job very easy, but it was not an available option for my "hacking" application.
> I configured the oscilloscope to collect 100M samples at 1 TSPS (tera-sample per second, 10^12) ...
Wow! Sampling at 1000 GHz? I wouldn't have thought that was possible! Does anyone know how the scope can even do that?
The higher sample rate is useful, because it effectively gives higher precision when finding the zero crossings; however, you could reduce the .wfm size by sampling at 50 GS/s on the scope then upsampling on the computer.
That would be better. Upsampling on a scope is dangerous, it's just too easy to fool yourself about what you're really seeing. Much better to use a slightly more clever algorithm on the PC than just "nearest crossing".
We tested the 5 Series predecessor to your 6 Series and I was amazed to see 10 or so volts on a 3.3V logic signal... until I remembered to check the interpolation setting. There was, in fact, no measurement above 3.3V (ok, within the usual tolerances) but the dumb shit machine had made things up and displayed them as if they were real. Even though they would have meant there was a serious hardware fault (not impossible, it was a new design going through bringup and there were in fact higher voltages on the board!), nope, it was the fault finding tool that was faulty.
I've had zero interest in the 5 Series since that demo week. Terrifying machine. (And it was supposed to have been "debugged" by that time.)
(Also, the probe itself has a max bandwidth of 10 GHz, so it will filter those frequencies before they're sampled by the oscilloscope)
For scopes that can do these high sample rates they frequently use two lasers and have the scope capture the beat between them as a demo of how fast they are.
The fastest scope I know of is a 256 GSPS done by interleaved ADC's. Analog bandwidth is 100 GHz Here's a recent submission on the bad boy:
Thank you!
One startup I was at had a important hardware component and it was a delight to work with electrical engineers
The physical realm is full of limits. Much of electronic design is about working within hard, physical boundaries (I was an RF engineer, so there were a lot of boundaries). For example, you could see the “ringing” in the scope traces (those little “bumps” near the corners of the voltage level transitions). Ringing can have serious consequences in real life, and things like track length, or even solder burrs, can affect it. Then, you have attack and decay, which can be affected by things like cable length. Basically, there’s no such thing as a perfect square wave.
Also, when you are running at GHz frequencies (not really an issue, back then), every solder burr is a microwave transmitter. That’s fun.
Software didn’t have these “fences.” I could go pretty much wherever my imagination took me.
I’ve always been a cantankerous bastard. I don’t like being told where I can’t go.
Knowing the physical realm helps a lot, when it comes to understanding software, though. I’m glad for it.
Go team Oxide, we're rooting for you!
It would be neat cause then you could wrap the decoding bits in a library that implement the libpcap interface. Then any existing tool like tcpdump/tshark/zeek would just work and it would effectively be a software defined network tap.
> ... this means we should catch 1-3 UDP packets. > After hunting down a USB key, I ended up with a 191M .wfm file to process.
assuming 3 packets, that's about 64MB of raw data per ethernet frame. The tshark output shows a 82 byte frame. That works out to 818400:1 overhead. Close to 1MB of .wfm data per 1 byte of packet data.
I suppose you could do some processing or compression closer to the input, but then you're just building a nic at that point.
Surprisingly cool going from raw IQ/analog data to ethernet frames much like the op has written.