tcpdump is amazing (2016)
jvns.ca
jvns.ca
Edit: Forgot the nprobe [2] (same author), allowing flow data creation and export, expecially where L3 flow capable devices are not usable (e.g. intra-VLAN)
HN really doesn't like or wants comments saying "agree", "nice", "+1" etc., because they add nothing to the conversation. Just upvote a comment/post if you agree or like it :)
K2289: Using advanced tcpdump filters https://support.f5.com/csp/article/K2289
OMG, yes, very well put. When I get a bug report with a pcap file I'm happy because I know I'll be able to see exactly what happened.
Speaking of which: for one of my libraries, I want to make a diagnostic tool that replays an interaction. My library mostly operates at the TCP level (also some UDP), so I need to reconstruct the TCP flows in my tool to feed to my library. Either I need an easy-to-use Rust library to do that directly from pcap files [1] or some format that represents bytes moving over the flow (like sets of lines with a timestamp, flow id, and pretty hexdump of the bytes) with a tool that produces it from pcap. This seems like something that should exist? Wireshark's "Analyze > Follow > TCP Stream"’s "Save As…", "entire stream", "hex dump" is kind of what I want, but it doesn't have timestamps, and it doesn't have a way to put everything (multiple flows, UDP packets also) in one file. Seems like there should be something under "Export Packet Dissections" [2] but I haven't found quite the right thing there either.
[1] https://crates.io/crates/pnet looks promising but it wasn't as obvious as I hoped how to plug it in for what I want.
[2] https://www.wireshark.org/docs/wsug_html_chunked/ChIOExportS...
tcpflow (mentioned in another thread here) seems much closer in that it does reconstruct TCP flows from pcap, but it doesn't timestamp stuff (it'd be nice to have an idea if some client->server data or some server->client data came first as well as just relate to timeouts), and I'd prefer to put everything in one file (both directions of TCP data, UDP packets).
Maybe I "just" need to figure out that Rust library I mentioned in the grandparent, and maybe create my own intermediate format that has just the data I want for my library. (I can discard TCP retransmissiony stuff, MACs, etc. to focus on what a library sees through the kernel socket interfaces.)
tcpflow stores all captured data in files that have names of the form:
[timestampT]sourceip.sourceport-destip.destport[--VLAN][cNNNN]
where: timestamp is an optional timestamp of the time that the first packet was seen
.B t
Prepends each filename with a Unix timestamp (seconds since epoch).
.B T
Prepends each filename with an ISO-8601 timestamp.This is probably part of the reason why Google is super fast and most software is super slow.
Same for IT teams. I should be able to assume that any answer to a network question includes the relevant info from the wireshark trace not just, "I can't see anything wrong with it"!
Otherwise, matching for TCP port, etc. fails to capture fragments which do not have the TCP header and the resulting file is missing some data.
When I was running a webserver with worldwide audience and 40+ Gbps traffic (most of that was our apk though), I saw no more than a couple fragments per second; unless there was some UDP reflection DDoS going on and I was getting fragments from that.
Lots of high profile sites don't even accept ip fragmentation because it's too costly to deal with.
Of course that sometimes fails, so if you still have TCP fragmented segments, the next best thing is to filter by source/destination address, saving to a PCAP file, then run tshark on that file with the "-2" flag which does packet reassembly.
(I don't know if tshark can be made to do on-the-fly reassembly, that would require keeping a buffer of un-reassembled fragments until the rest of the packets are seen.)
e.g.
> wireshark -k -i <(ssh -l root remote-host "dumpcap -P -w - -f 'not tcp port 22'")
Have your prod boxes set up with proper permissions for sure but there's hardly a practical risk doing this in dev/testing.
Run nothing as root, unless absolutely required. And this had nothing to do with C or rust, running rust based software, any software as root, when not required,is unnecessary, and risky as well.
Nothing is safe. Behave that way, or you do your oeg, and yourself, wrong. And its not OK to treat a dev env as fine to get compromised.
Many orgs get borked by someone taking over dev machines, or infra, and sliding in that way.
Good advice, to be sure. Especially anything getting network input[1].
In a reddit thread once, I pointed out that there is no need for any server/service program to have any access to the file holding the startup configuration. There really isn't.
If the user needs to write that configuration, use a different program with elevated privileges that do nothing but write that file.
When the server starts up and needs to read config, it should start up as a user with elevated privileges, read the entire file in the first 3 lines of `main()`, then drop privileges and continue execution as normal. It will never be able to access that file during the rest of its execution.
I don't think I've ever had a comment downvoted on reddit so hard!
[1] No need to say "untrusted network input" - all input is untrusted unless you are literally in control of both parties.
It would add fair bit of code, sure, but nicely isolated and not complex: It's three extra lines of code in the main function, and a separate utility to write the config.
Balance that against the fact that the server then never writes its own config - that code is moved into the configuration-writing application, which removes complexity from the server application.
> and in particular couple the application to a specific combination of operating system/privileges model/file system.
It does indeed couple it to those systems that are sufficiently POSIX-like; however if you are writing a server that doesn't run on a Linux or BSD-derived system, you are out in the left field anyway.
> How great is the benefit, really? How many things are exploited by the application writing to its own startup config?
Well, the comment I made was in a thread discussing a vulnerability in the Microsoft Teams application[1] which was exploitable to retrieve the user's secrets[2] from their config file.
So, that particular comment was very much on-topic and in-context.
[1] Not sure if it was on all platforms or not.
[2] Or something - I forget exactly what was leaked.
How much is probably going to depends on how everything is configured, what your threat model is, what service it is, and a million other things.
If a malicious actor compromises your normal user account, they can also compromise your configuration files and alias sudo or set up a keylogger or do all kinds of nasty stuff.
Going from standard user to root isn't much of a challenge for all but the most minimal infections in almost every standard setup for common Linux distributions.
Malicious hackers don't need anything other than your current user permissions anyway; your API/ssh keys/crypto wallets are all stored in your home directory or other places a malicious program can get access to.
There are exceptions, but I very much doubt that most people develop on QubesOS levels of Linux security.
There is no practical antivirus software on Linux that will catch anyone sophisticated enough to exploit tcpdump. You need a LOT of know-how to run a "safe" Linux system that you're more likely to learn as a sysadmin than as a developer. Security is hard, especially on the Linux desktop, and it will be as long as attempts to add it are met with responses like "they're trying to take our freedom away".
Common internet resources aren't much better ("just disable selinux") and current solutions for usability problems in this space (i.e. sandboxed applications not being able to access the directories you chose because they don't correspond to what the dev expected) aren't very great either.
I definitely think we should use the resources for secure desktop computing that we do have, which includes running tshark and friends at the lowest level of privilege possible, but I can definitely understand why people ask "why should I bother" when most of their setup runs at Windows XP levels of security out of the box.
I mean, are you really running tcpdump on your local computer? I imagine you would be running it on the server you are trying to debug. So the security setup would plausibly be a bit different than that.
But generally i agree. Good practise not to run things as root, especially this type of tooling, but its not exactly putting your private key into a public git repo level of insecurity by any means.
I did several times when I needed to debug or record some traffic because I couldn't figure out why some application wasn't communicating right. Wireshark quickly got overwhelmed with packets so I used tcpdump instead.
I'm most likely running tcpdump on (near) production servers because it's often a tool of last result, but sometimes it's just the right tool for the job (or just as good a tool as the fancier ones, and why not stick to the universal solution?).
Sure, the attack surface of tcpdump is much smaller than wireshark, so the issue is not as pronounced, but wireshark indeed had a lot of vulnerabilities in the past: https://www.cvedetails.com/vulnerability-list/vendor_id-4861...
It has tons of dissectors which aren't nearly as battle tested as my network stack.
I'm also going to add that I have passwordless sudo, so there's no meaningful difference in my case.
Assuming you care about infosec from your profile, privilege escalation is definitely something you want to avoid especially if you're using these tools in for example a CTF engagement where the first thing I'll be doing is enumerating my adversaries
other dev: this is insecure
me: what's your threat model?
other dev: <convoluted, opportunistic-only highly-timing-sensitive scenario that assumes attacker has gained internal network access>
me: you do realize all our boxes have an "admin:password"-esque standard login available and passwordless sudo. if they're on our network they can already just do basically anything.
People (well, devs) love to think about security in very fine-grained detail, and honestly that's good. We should all be thinking about this. But your security is only as good as your weakest link, and when your security policy already literally codifies "we will trust the firewall to make intranet security less annoying/complicated" then unless you want to go change that or your threat model is describing an attack on that security model, then please don't try to get overly clever.
Now - that said - I do wish more orgs just did defense-in-depth from the beginning. It's so easy to provision certs nowadays that there's almost no reason not to do mTLS etc even in something like a homelab setting.
recent example: used tcpdump -X to debug why my wireguard setup wasn't working, since with UDP you just kind of get a shrug as to why or at what hop the udp packet got lost or filtered. So I ssh'd to all the middle boxes and just tcpdumped the UDP ports I was interested in.
https is only one protocol. There are hundreds or thousands of other protocols that tcpdump works fine with. Some of these protocols are underneath https, including tcp, which is in the name of the tool. So while tcpdump may not function as a tlsdump and can't see into your tls connection, it does a lot of other things great.
You may as well complain that your screwdriver is no good for cutting wood. Clearly you need a different tool.
with everything moving to https, sometimes even local environments, it's basically been useless now however.
It won't apply to every situation, but might provide some building blocks you can adapt to your environment.
But yes, tcpdump and strace/ltrace are the ultimate tool for "I have no idea what the problem is, let me find out exactly what is happening" when troubleshooting.
[0] https://www.netresec.com/?page=Blog&month=2020-01&post=RawCa...
And the ability to do that type of troubleshooting when the shit hits the fan is becoming more and more rare. I regularly run into devs, even so-called “senior” devs, these days who don’t know how it’s useful to look at packets or system calls and whose minds are blown that anyone would know details that far down the stack. It would be depressing if it wasn’t so good for job security.
I wish the same thing was available for i2c and uart reads, for example.
It's a great way of finding shitty network devices like Juniper SRXs which just randomly and silently drop packets in the middle of flows.
Tcpdump is amazing - https://news.ycombinator.com/item?id=11302992 - March 2016 (102 comments)
Filter rules syntax is here:
> sudo tcpdump -n udp port 8125 -v -X -i lo