It's probably not Rails itself...
I had a similar issue, which turned out to be a bad Rails plugin, which maintained a few unnecessary pointers to my model classes. The small plugin caused a 20+MB per request leak in development. I rewrote the piece of functionality I needed from the offending module & the issue went away entirely.
Sadly, there isn't really decent memory profiler support to help you track it down. I found the issue by running `git bisect run` with a little script which restarted the server, beat on it a bit with `ab` and then printed memory usage with `ps`. Took me about 10 minutes to track it down to the exact offending commit.