Ftrace: The hidden light switch | Hacker News Reader

Ftrace: The hidden light switch

lwn.net

145 pointsluu10 comments

Slightly OT, but the I/O inflation caused by raising read ahead to 2048K seems to be pretty huge. For a DB server that might only be reading a few K, I can see why it caused an issue.

Anybody knows why Ubuntu did that? I would have expected that in the age of SSDs, read-ahead becomes less useful.

I don't think he says anywhere in the article that Ubuntu default was to use 2048Kb RA. On my 14.04 LTS installation all block devices are set to use 128Kb by default. Maybe they were experimenting with it after the upgrade.

In 2014 this should all really be auto tuned - how hard it is for a user space tool to figure out that a disk is SSD, do a seek latency and read speed test to conclude it will be better off without RA? (Thinking of it though RA brings data in RAM which is still orders of magnitude faster than reading from SSD. So may be there is _some_ benefit if you don't increase the IO overhead enough.)

Then I might have misread that, sorry - 2MB would have been really surprising as a default. I fully agree about the auto-tuning!

Wow great debugging article from the performance guru B.Gregg. I'm also not comfortable tracing kernel functions as of yet and remain in user space so far with Sysdig,strace,ltrace.. but a tool such as perf(1) I definitely need a more practical understanding of.

Here are some perf(1) highlights if you want to get started:

• https://news.ycombinator.com/item?id=6842338

• https://news.ycombinator.com/item?id=7679822

• https://perf.wiki.kernel.org/index.php/Tutorial

Try to look closely at the methods used to correctly apply perf(1) to the problem at hand. If it feels like you understand _why_ it was used, you'll easily be able to determine if it's the right tool for the job.

iosnoop (a shell script using ftrace to implement 'tcpdump for your disks') was introduced here last month: https://news.ycombinator.com/item?id=8045632

Iosnoop for Linux you mean, the first DTrace-based version of iosnoop for Solaris was written by Brendan over 10 years ago.

The article was really interesting for my developing work, but at the final conclusion I felt as if he oversold it as a tool to diagnose system problems: The issue was that he did not restart the database after tuning readahead, right?

So instead of the "have you tried turning it off-and-on-again", which takes next to no time, we venture off into tracing kernel code?

Read ahead is a kernel level parameter; having tuned it before (also for a Cassandra ring) I would not expect to have to restart any user level processes for it to take effect.

But the real issue is understanding the platform that you are building on, and having the tools to explore how that platform works. I am so thankful that Brenden is out there doing this work and sharing it for the rest of us to benefit from!

I thought Netflix was an all-FreeBSD house. Good to see they have some Linuxen too.

Reply on news.ycombinator.com