Splitting the ELF sections into multiple chunks is such a good idea.
It'll probably have a performance hit if you have to use functions from other chunks (need an extra lookup in the table to find them) but if the linker is intelligent enough you could avoid a lot of that by putting stuff that calls each together.