I can't remember where I saw the numbers and I'm pretty sure I'm off a bit. But I think only half of all crude oil gets distilled into "fuel". The rest ends up as plastics and other chemicals, as you mention. I just remember it's a very significant of non-fuel oil use is out there. A quick google search, I can't find the chart I saw. Maybe someone can chime in better on that?

Edit: Listen to Retric's comment below, not me on my stats