I found this by looking at spark source code to figure out what they were doing under the hood to deal solve this problem. :)