Linux debugging tools you'll love | Hacker News Reader

Linux debugging tools you'll love

jvns.ca

400 pointsrollulus63 comments

I think it's a good contribution. Couple of things though:

- lsof can also print the list of files open by a process. fuser can tell you which process open a specific file. lsof is mentioned later in the zine though.

- sysdig is not mentioned but is great.

- wireshark/tcpdump/ngrep/etc might be challenging to practically use in the presence of SSL/TLS. you can mitigate this problem by setting up SSL termination at your network perimeter so you can monitor unencrypted traffic.

- additionally to netcat you have curl, httpie and many other clients for similar purposes. in the Chrome developer tools network tab, you can select a request sent when visiting a website and click "Copy as cURL". this will quickly export any request you were making as a cURL command. likewise, proxies like burp also implement this same feature.

sometimes you need to debug DNS problems, that is not covered sadly. I use dig for this purposes but I wonder if there's something better.

if you are going to be using command line utilities, getting acquainted with cat, grep, cut, sed, awk, tail, head, colrm, tr, sort, uniq, comm, wc, etc. is very recommended. ministat is also cool.

then, about java and node... java has excellent profiling tools, including the free jvisualvm tool shipped with Java that does the job for the most part. profiling and debugging node in runtime can be really challenging. specially analyzing node coredumps with mdb_v8 is not for the faint of heart... you need to set up a VM with joyent's SmartOS for this. there's an npm package that simplifies this process called "autopsy". now, flamegraphs might be fine, but i strongly prefer nodegrind + qcachegrind.

> wireshark/tcpdump/ngrep/etc might be challenging to practically use in the presence of SSL/TLS. you can mitigate this problem by setting up SSL termination at your network perimeter so you can monitor unencrypted traffic.

https://mitmproxy.org/ is pretty awesome for that type of work.

In Firefox/Chrome (when using NSS), on developer builds (Aurora/Canary), you can also set SSLKEYLOGFILE to a path and use Wireshark directly.

https://wiki.wireshark.org/SSL#Using_the_.28Pre.29-Master-Se... https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NS...

Additionally, Chrome offers chrome://net-internals which let's you inspect a good deal of information about the request being sent.

> ways I've changed how I think about debugging:

> bug is happening for a logical reason. there's no magic

> be confident I can fix it

> talk to someone

Those are good steps. IMO "talk to someone" is a critical step but you'll get wildly different results based on the quality of your reference resource. Ms Evans likely has access to someone (or is someone) who clearly knows their stuff. eBPF and other debugging features enabled by modern kernels are not well known by many developers.

But "be confident I can [diagnose] it" is a great first step and if you're persistent you will find the help you need. Sometimes it may take plumbing the depths of SO or IRC to find that help, but it's out there.

Finding someone smart to talk to is important, and a lot of people fail at this step. I like to consider myself decently good at Linux troubleshooting and get asked a lot of questions most days.

My sole piece of advice to folks looking for help: Be specific and help yourself. This means don't come and ask "why is my server slow?!" - it's coming into the conversation telling me the exact symptoms and what you've already tried and where you think the next steps are.

If you are not at that point, you haven't put enough effort in for me to bother. Those are called paid consulting engagements.

I would say most people get this wrong, and end up being ignored over time. The few folks who consistently give me interesting (even if they are trivial and I've seen it before) problems to help them with that they just need a bit of specific knowledge to solve? I look forward to them contacting me.

You don't even necessarily need to be talking to someone smart, or with any knowledge in the subject at all. Often times I'll figure something out just by explaining the problem to someone. The act of breaking down where the problems could be and showing someone what you've tried often leads to more ideas on how to fix it.

hence https://en.wikipedia.org/wiki/Rubber_duck_debugging

Interesting. Somehow i have no recollection of that term even after reading that book. It was a number of years ago now... perhaps it's time for a revisit.

Only for problems that were just out of reach for your normal mindset but still ultimately within your capability to handle. At a certain point, you need someone better than yourself to get better.

Yeah.. I've solved quite a few problems by writing out a question on Stack overflow but found solutions/ideas before posting it.

This is good advice, and understanding this is the key to interacting with 'hacker' types.

A good document to read or link from your project's "Contact" section is this FAQ on asking smart questions:

http://www.catb.org/esr/faqs/smart-questions.html

It might not be quite right for your project but for many projects I've been part of it's pretty spot-on, at least as a starting point to avoid the worst of the 'bad' questions. Our IRC bot had a command for linking this to people, haha O:).

Related: http://bash.org/?152037

> Write out a good logical question for stackoverflow and answer my own question.

90% of the time!

The cutesy/manic writing style is a little jarring, but it seems like a lot of effort was put in to making a useful resource. Worth taking a peak at, don't let the style scare you off.

I like the change of pace. So many of these types of guides seem like cookie cutters of whatever the latest writing and design trends are. I find it nice to see indie zine style (stuff I like to read on my off time) fused with a practical technical resource (stuff I like to read on my on time).

To be fair, this is a spin-off (or elaboration) on an earlier post, where (if you follow the Author's blog) she specifically talked about how she'd like to write her own zine. The style is the point of it, I think.

If you want less cutesy, here's the related post: https://news.ycombinator.com/item?id=12059156

The approachable/playful writing style makes it so much more engaging for me. It's refreshing to see some of this stuff explained in a conversational and enthusiastic manner. Obviously, just personal taste.

You weren't kidding.

The content is good (for the 7 pages I was able to follow it), but the comic book delivery is not working as well as her previous blog posts. Not searchable, more difficult to follow.

This 'zine (and really, much of the author's entire website) are pretty much pure, concentrated joy and tech-geek happiness. I hope my kid grows up to have the sort of attitude about solving problems and going about life that she does.

Bravo, Ms. Jvns!

While we're on the topic of simple debugging tools. One I find I use all the time is top. Not classically a debugging tool but it sure solves a debugging problem and I find myself using it very often to check CPU and memory usage.

On the topic of more sophisticated debugging tools I like valgrind and Google's gperftools. These are definitely more difficult to use but I recommend not giving up on them because the pay off is huge. The trick is a) knowing how to run these tools and b) knowing how to read and understand the output. Both can be achieved through practice and RTFM.

I prefer htop to top (or glances if Python is an option), but it's true it's not installed by default.

Atop is also a great tool

Regarding "valgrind" -- I've had better results with electric fence and ASan/TSan/UBSan, especially since most of them work on non-x86. And I've had issues with valgrind when investigating a program that leverages newer x86 instructions than it was built to interpret.

If you find this interesting, you might also like the embedded.fm podcast she was on: http://embedded.fm/episodes/141.

Anyone have more details on the netcat file transfer trick on page 9?

You would want to confirm the data received matches the data sent somehow right?

You start a listening nc on the destination server, and push the file from the source server.

    Destination: nc -l -p 1234 > foo
    Source: nc destination 1234 < foo

Protip #1: You can also send directories:

    Destination: nc -l -p 1234 | tar xf -
    Source: tar czf - directory | nc destination 1234

Protip #2: pv for progress indicators (pv on one side is enough)

    Destination: nc -l -p 1234 | pv > foo
    Source: pv foo | nc destination 1234

Note: netcat (BSD or GNU variants) syntax varies across unixes and distros. Sometimes it's `nc -l -p 1234`, other times `nc -l 0.0.0.0 1234`. Check your man.

You can check for data corruption with md5sum or similar checksumming tools.

Sysdig was not mentioned.

Most of all those tools / command, can be achieved with this one tool / command.

http://www.sysdig.org/wiki/sysdig-examples/

This is by far best enjoyed on paper. Print one out and share it with your office!

This is awesome !

Wow, that is a different style than I'm used to. Must say I'm happily surprised by good content, despite the playful look. It makes it seem very accessible, nicely done.

Just wondering if she draw/wrote everything by hand or there is font like that?

She wrote about it not long ago: http://jvns.ca/blog/2016/08/29/how-i-made-a-zine/

Short answer: she did the first zine on paper, and the second one on a tablet.

I am aware of this R package for generating xkcd-style plots and text: https://cran.r-project.org/web/packages/xkcd/vignettes/xkcd-...

Actually there is "real" font made from xkcd https://news.ycombinator.com/item?id=9302740

I appreciate the effort and enjoyed this paper. One small remark: the first example is ridiculous. You don't need such a low level tool to find out someone is doing crazy database queries, just analyze your slow log!

netstat is obsolete.

What do you recommend as a replacement? I'd love to get something that wasn't quite so opaque.

Ss has replaced netstat. (By the way each time you're typing netstat you're suppposed to get a message telling it's obsolete). ss is an improvment on performance and features, but unfortunately the ui isn't much improved compared to netstat ...

> Ss has replaced netstat.

what about non-tcp/non-udp sockets e.g. sctp ? ss doesn't seem to support that (but i might be mistaken)

You're right but as I'm not using it I don't know what's the situation about that.

I like ss, except I can never seem to remember the name. The two letters don't seem to have any relation to the functionality.

The man page refers to it as "ss - another utility to investigate sockets". That doesn't seem to help.

For different (historical ww2 reasons) I don't like the name and I wish ss had a more meaningful name ...

ss - Sockets Stat (?)

In Europe ( at least in France ) ss means Nazis' Waffen SS. Something like sostat would have been a better choice imho.

Sorry, I'm from Europe too, and I did feel in my skin, the Nazi ideology that remains, when working in Holland/The Netherlands for 2 years. If I read "ss" in a computer/programing context, I don't think about Nazis. The same that I love rainbows, and I'm not gay.

People takes symbols as a totalitarian thing. We are free to skip their imposition and use things as we like. We are still free to do it. Really. We can appreciate the beauty of a rainbow after a rain/sun combination, and not impose our sexual unsolicited demonstrations to anyone because of it. It's just a nature thingy.

We can say "ss" for a program that print sockets stats, and skip remembering Nazis. It's opensource, any skin tone can use, modify and redistribute, ss. We can do it, or, we can stick to bad memories, and name things just and only, as Israeli stuff.

I didn't think about it, and probably the person that did name the program, neither. Anyway, I see your point, the name can bring negative sentiments to some persons which see the daemon in a piece of source code or in an acronym in unrelated context.

Millenials in my team think like you so I think It' s not that bad. And for some time I was a a hudge advocate for a js testing library named "testacular" which is far worse in retrospective ... " What's in a name" ...

Nope, sorry. I'm with the (great grand) parent(s) on this one: my first association when reading about 'ss' in this thread was "Schutzstaffel" as well. I think I would think about it every time I'd have to type it into a shell, but luckily, I don't do much network related development.

Nifty, `ss -i` seems useful. I think most of the rest of the info you can get with netstat, though (and there doesn't seem to be a way to resolve usernames, as `netstat -e` does).

Is there a way to make the output not justified to the width of my terminal? The extra whitespace makes it hard to see which rows match up, and also makes it annoying to paste into IM or email.

> (and there doesn't seem to be a way to resolve usernames, as `netstat -e` does)

Challenge accepted:

  ss -e | grep uid | gawk 'match($0,/uid:([0-9]+)/,u) {printf "%s user:",$0;system("getent passwd "u[1]" | cut -d: -f1");}'

Not your point, I know, but it was fun.

You lose three points for using grep and gawk in the same pipeline. :P

Oops, the grep was just to filter the lines that actually have a uid while I worked on it, it's not needed or desired if you were to actually use this. You can remove that part. :P

How do you get `netstat -s` information out of ss?

> By the way each time you're typing netstat you're suppposed to get a message telling it's obsolete

Starting about when? Most of the distros I use are a couple years old, and I don't see "ss" under the package managers.

It's installed by default at my Debian stable computer. Yet, there is not package for it, must be at the basic tools.

It's from iproute2 - sockstat "ss" Circa ~ 2001, although it's been through a few maintainers since.

https://github.com/shemminger/iproute2/commits/master/misc/s...

I wish utilities would use more characters, Because it makes searching so much easier. You can always alias it to something else, but for example, it is way easier to search for ifconfig than it is to search for ip (for iproute)

AFAIK ss was not meant to be a complete replacement of netstat. The ss utility (part of iproute2) uses netlink to monitor sockets only. The netstat utility (part of net-tools) uses old school ioctls to do things like manage routing tables, multicast membership, interface stats...long story short, iproute2 replaces net-tools.

ss -anp | grep LISTEN

You don't need grep for that, ss can already filter by state on its own. Use: "ss -o state LISTENING"

Sometimes I find typing grep/cut/awk/etc to be easier to remember than custom flags and thus faster to type. Often times my time spent looking through the man page is better spent just writing a more verbose command line.

+1. You can see the same effect in natural language as modern English has fewer tenses and declensions and makes heavier use of helper words, as contrasted with olde English. Same with Latin vs modern romance.

care to elaborate?

Now you cat get open socket list in more flexible (c), robust (r), cleaner (tm), optimized (gmbh) way than 'proto-src-dst-state'. You just have to reread an entire man page and remember all new command line options burden. After that, your productivity will explode. Again.

Reply on news.ycombinator.com