Are you explicitly using a linked/embedded font resource, so that you control exactly what font is used to render the text? If not, there's certainly no expectation of uniformity.
Maybe if we had container queries, and could set font size as a…container unit? I don't even know what to call it. Like vw and vh units, but relative to a parent container.