I mentioned in another post, and as someone else also mentioned, that it would hypothetically work well if you kept the sizes at multiples. Ie i wouldn't want to break the grid and break alignment, so vertically a smaller font size would go in multiples, such that each downward increment of font would be half the size.
Makes it a bit more complex though if the block sizes make subdivision like this difficult, though. This isn't an idea for an implementation necessarily, rather just an idea to illustrate my request. I want to have larger font size than my gutters/etc in my term editors.