My long-standing observation has been that while nature may abhor a vacuum, she also really, really loves sigmoids.
That performance vs. training data is not linear, but logarithmic, doesn't exactly come as a surprise.
That performance vs. training data is not linear, but logarithmic, doesn't exactly come as a surprise.
I suspect from the success of Phi3 that it is in fact the former.