Tcpdump Examples

Using Tcpdump to analyze HTTP won't always work. Unless I'm mistaken, Tcpdump doesn't decode packet fragments into a stream, so if the request is fragmented, your grep won't work. If there are multiple HTTP requests in one TCP stream, it's less likely they are aligned completely in one packet, also fragmenting the request. And if it's a request to Google it could be in QUIC rather than TCP. You may wonder what on your network is broken because you can't see the requests, but it's just the network fragmenting packets.

Tshark (nee Wireshark) I believe does decode packet fragments into streams, and allows you to use more advanced Wireshark protocol filters. My suggestion is to start a pcap capture, and at the same time replay it using Tshark with the filters you want. This way you can re-analyze the same live traffic both now and later.

I use tcpdump continuously to analyse multiple RTP feeds coming in to our edge network. Found all sorts of issues with it, including packet reordering in Cisco and mikrotik routers (shouldn't be a problem with rtp - but tell that to ateme decoders), actual packet loss on juniper srx, and of course minor outages on international and even local links on the order of 100-200ms of loss.

Packet stream is dumped into some perl which keeps across the sequence number looking for out of sequence packets, summarise output every second into an elastic stack. Also have it comparing multiple streams for dual streaming and comparing skew.

Ironically I rarely use tcpdump to look at tcp packets - about the only interesting thing to me there is the SYN packets to see if the packet is getting through the firewall!

I think a similar post on leveraging Wireshark would be neat. The deepest I tend to need to go is searching packets for a known substring and then following the TCP stream to see where something unexpected happened. Add to this the analyzing of the frames on this stream and you can diagnose tricky timeout issues, bad protocol usage, and much more.

I recently presented my tool HTTPanalyzer at RIPE76. It's a C tool I developed in 2016 available at Github for processing HTTP traffic up to 10Gbps from PCAP files or straight from the interface. I recommend the branch "revisited", much better coded in my opinion. Of course, it's limited to the first packet of the request and first packet of the response. It's aimed to aggregated statistics like response codes, user agents, response time (immediate one, not full load of the resource) etc. More info here: https://carlosvega.github.io/httpDissector/

Only HTTP traffic, you can process decrypted HTTPS traffic (like some devices do, i.e. IXIA network devices) which is transformed into HTTP traffic. Regarding HTTPS or HTTP2 etc. The current approach is to correlate the application information from log events against traffic measurements.

It's pretty sad, that the best way of analyzing HTTP traffic from the command line appears to be comparing payload bytes:

  ~$ sudo tcpdump -s 0 -A -vv 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420'

There should be a better way to do that. Ideally I would want a tool shows Request Method, Headers, Query String, POST Payload of requests as they come in (and let's me filter on those). It should support HTTP2 and know how to stitch together the payload from multiple packets.

HTTPS is a difficult topic. I think it's fair for a tool like this, not to mess with encrpytion. Usually you don't want to had a precious private key file to a command line tool for debugging. A better way seems to be to terminate SSL on a separate host, and analyze the un-encrypted traffic.

My tips:

1. Use `-i any` -- just in case the traffic you want to look at is not on the interface you think it is. Also doesn't put the interface(s) into promiscuous mode, which can be preferable.

2. Read the `pcap-filter` man page.

Good stuff, thanks for sharing. I always forget you can do indexing into the packet itself as in tcp[((tcp[12:1] & 0xf0) >> 2):4].

Also sometimes I reach for ngrep if it is installed:

https://github.com/jpr5/ngrep/blob/master/EXAMPLES.md

Shameless plug, but if you’re interested in this sort of thing, I published a proof of concept docker container that makes packet analysis with scapy very easy. [0] The trick is to leverage iptables nfqueue to filter packets you’re interested in, and have scapy listen to the queue.

[0] https://github.com/milesrichardson/docker-nfqueue-scapy

Nitpick: The HTTP examples will only catch v1 traffic, not HTTP2

The "-nn" explanation seems wrong. Neither libpcap's tcpdump nor BSD's need more than one "-n" to not translate port. Even in their examples, we can spot that using a single "-n" do not translate port names.

Additionally, it's always better to always use "-p" if you don't want to accidentally attract additional traffic to the host (additional VLAN on a 802.1q interface, or additional multicast traffic).

NetEng here, I'm usually troubleshooting lower layer issues (Ethernet, MPLS, IP) than HTTP however, tcpdump is one of my go-to tools every time.

If anyone is interested I've made some notes here: https://null.53bits.co.uk/index.php?page=tcpdump-notes

For example, imagine spanning/mirroring a 10G backbone link, how many people are pinging 8.8.8.8 all the time. I can ping with a specific DSCP value set to isolate my pings from anyone elses, looking into the reported issue of 8.8.8.8 latency, then apply a filter to tcpdump on my mirrored port that matches ICMP traffic to a specific IP, with a specific DSCP value, inside specific L3 VPN (specific MPLS labels) etc.

sudo tcpdump -nlASX -s 65535 -vvv -i eth3 '(mpls 52634 and (ip and (ip[1] & 0xfc) >> 2 > 0x01) and host 11.22.33.44 and icmp)'

I love tcpdump's filtering capabilities.

I am always amazed at how many people have heard of tcpdump but how few seem to have heard of tcpflow

https://linux.die.net/man/1/tcpflow

If you are looking to do TCP level protocol analysis it is one of the simplest and easiest linux command line tools out there.

It's biggest strength is that it can take a payload split over multiple packets and spit it out as a file with the whole payload

tshark is supposed to be a tcpdump alternative but it seems tcpdump is still dominant on the command line these days.

I can't count the number of times tcpdump has made debugging a problem a zillion times faster. It's my go to tool for any network related issue.

I tend to end up using wireshark for analysis of more complicated SIP issues but I always start with tcpdump.

tcpdump + CloudShark is my go-to. Usually using cURL with their API or SCP via Couchdrop.

how practical are examples ignoring https?... or am I mis-understanding target use case?

There is a whole industry for distilling manpages down to stupid chunks and making blog posts it seems.