Ah, lat_mem_rd, so awesomely named :)
Yeah, that's mine and I think I was the first to do it that way though I saw the idea described in Hennessy & Patterson and they didn't give me credit so they must have dreamed it up too. It's clever but pretty obvious once you think about it.
Very cool that you are still using it. lmbench has aged pretty well, mhz.c still works. BTW, lat_mem_rd has changed processors all over the place. They all have code to detect sequential strides and prefetch. I think Carl and I switched it to go backwards to try and defeat their prefetch. Dunno if they prefetch that way as well. The prefetch is sort of cool and sort of annoying because it hides how the hardware would perform if the access was random. Without the prefetch you can determine cache line size, L1,L2,L3 size and latency, and TLB size, and main mem latency. With the prefetch you think you are getting all that stuff but it looks different. If you have ideas on how to defeat the prefetch and still get the info I'd love to hear them.
Another funny thing about lmbench: I was recently approached by a company that's got it working on phones. They want to feed back their stuff and have me do another release that makes it easy to run it on phones. I might do that, still waiting their dump of code. Kind of a neat idea.
I still tinker with lmbench, if you have something you want measured and it fits with the idea of measuring bandwidth and latency of everything (and fit's with what Linus and I wanted out of lmbench, if you tune for those metrics you are making the hardware/OS better, so no silly show off your whatever, has to be generically useful), let me know what it is and I'll see if I can code up a benchmark. Making new measurements using the lmbench framework is pretty easy for me, I know that code.
BTW, I have an internal version of lmbench that adds stdio support to lmdd (if you haven't played with lmdd, man you've missed out. It can simulate many I/O benchmarks and give you results really quickly).
So why would I want stdio support? Because BitKeeper has an enhanced stdio library that can stack filters on a FILE*. And it has some interesting filters, like gzip, lz4. And some much more interesting filters that do CRCs on blocks and an XOR block at the end. And another one that is so complicated I would want to describe it in person.
You can combine lz4 with the CRC stuff. Once I had stdio support, I linked with BK's stdio instead and added options to push all those filters and I could see what the overhead was. I did all this when Wayne Scott was putting that crud in BK so I could make sure we were not slowing BK down (we weren't, I think all that crud runs around 1 GB/sec which was fast enough for me). Oh and I also added support to do I/O backwards because we did that in BK as well so we had an append only file format for the ChangeSet file. Lots of funs stuff in there, I need to package it up and release it. If that I/O stuff is of any interest to you, it's open source under the Apache v2 license and if the license bugs you tell me what one you want and I'll rerelease it under that one. We picked that license because we thought it was the one that was easiest for everyone to use, if we're wrong we'll change it.
Nice writeup on the migration stuff, sounds like you are having fun. Getting paid to have fun is sweet, that's what it was like for me at Sun in the SunOS 4.x days. I got there and spent the first 3 years telling people "man, I love this job so much. I'd work here for free if I had the money." Someone took pity on me and said "Do you never want a raise? Because that's how you never get a raise" :) I shut up and started getting raises.
Enjoy that job. Not every job is that fun, you are at a special part of your career, enjoy the heck out of it.