Thank you, that is really constructive! I do agree with you on using ByteStrings everywhere (some laziness on my part). Calling it Data.Map is a typo, I will update it right away!
I was not aware of the benefits of fromListWith, gonna look into it right away.