> I'm reluctant to add asset pipeline
Consider using different/manifests layouts if you're concern is client-side. If it's compiling assets I think your concern is misplaced.
> admin panels tend to use large amounts of RAM etc
I don't think this is the case, or a huge concern depending on you're environment. Further, you could separate the apps into engines, they're great for this. TaskRabbit did a write up on this: http://tech.taskrabbit.com/blog/2014/02/11/rails-4-engines/
Personally however, I've normally roll out the app as one piece with active-admin, generically speaking most apps have little admin-specific logic to them, for example, it's normally something like: "Do <x> for user(s)", "Approve <x> content", the type of operations you do can easily fit under the simplistic nature of what active-admin offers. This isn't a catch all however, some applications require more of an integrative approach, in which case it's development as normal, bits of functionality injected here and there so-to-speak.