Adding a point to a size can give you a corner of a shape. You can have functions for scaling to aspect ratios, normalizing, and stretching. I don't see why any would be specific to one type or the other. In math you don't usually deal with points or sizes, you deal with vectors.

But admittedly conversion between Points and Sizes is not a pain point for most people. It's much worse when dealing with types that have the exact same purpose, just from different libraries.