Hacker News Reader

Comment by tedd4u | Hacker News Reader

In image recognition, Google asserted in 2017 that they were unable to find decreasing returns -- "Performance increases logarithmically based on volume of training data." Maybe for supply-chain and regression models there is a limit but it seems in deep neural nets maybe there's a different answer.

Blog: https://ai.googleblog.com/2017/07/revisiting-unreasonable-ef...

Paper: https://arxiv.org/abs/1707.02968

View on HN

Isn’t that the very definition of diminishing returns?

Perhaps it would be more accurate to say that marginal return is materially positive for all feasible n

A logarithmic increase in performance is positive for all n, but not necessarily _materially_ positive for all n. In fact there will be an n where the marginal gain from adding more data will _not_ be worth it, if the cost of adding more data scales linearly but the benefit scale logarithmically.

Right. If the cost of adding more training data remains flat or increases, a logarithmically increasing n will eventually reach a point such that it's more expensive to continue increasing training volume.

In fact, unless the cost of adding and using training data exponentially decreases over time, it's a mathematical certainty that a logarithmically increasing n will quickly incur expensive, diminishing returns for using more data.

So in the context of this Google paper, you could conceive of a situation where training data actually becomes easier to load (albeit subexponentially) and still becomes too expensive to use relatively quickly.

As a scientific term, "diminishing" means its derivative has limit/asymptote of 0 or negative approached fast enough that the integral (or series sum) is bounded. Logarithm, with derivative 1/x is the well-behaved function with smallest (ignoring constant factors and trivial higher order terms) non-diminishing return -- its returns flatten too slowly to bound it's value.

Yes, precisely. Logarithmic growth asymptotically decreases - it's the definitional opposite of exponential growth. This is clearly illustrated in typical graphs depicting logarithmic growth: https://jamesclear.com/wp-content/uploads/2015/04/logarithmi...

So in point of fact, according to the cited paper Google is asserting there are diminishing returns to increasing the volume of data.

The two of you are asserting different hypotheses than the OP presented: diminishing returns != decreasing returns. The Google paper found that increased amounts of data always improved performance, but did so at a lower rate the more data that had already been provided. First derivatives vs. second.

The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns.

The headline isn't consistent with the article itself, which states (somewhat confusingly):

* As the number of product data grows, the benefits were negligible

* More observations per product was important

* The results were consistent with asymptotic theory (central limit theorem) that predicts that more data has diminishing returns.

In any case, this seems to be a case of "diminishing returns."

I'm only talking about the Google paper brought up in this thread, because logarithmic growth does asymptotically decrease over time. I didn't say that the Google paper asserts absolute decreases over time, such that more training data actually makes results worse. Just that more data becomes too expensive to be worth it in a logarithmic context.

>> The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns.

Well, if I was paying $10,000 for 10,000 examples (to collect, cleanup, process, train with, etc), getting 90% accuracy and making $90,000 from the training model, and now I'm paying $10,000,000 for 10,000,000 examples, getting 91% accuracy and making $91,000 from the trained model, I'm losing money where before I was making some. That's "not better".

Few big tech companies pay for training data - it usually arises organically out of usage data from their products. You need to collect this anyway for product function / business metrics / abuse prevention, and build the cleaning & processing pipelines. So the only marginal cost of feeding more training data into your machine learning pipeline is the computational cost of training it, which is usually tiny fractions of a penny per sample.

If it was so simple to collect, process and train with (very) large amounts of data, everyone woudl be doing it. Instead, it's just a few very large companies that can do that, Google, Facebook et al.

Anyway, the cost per example doesn't have to be astronomical. If you need a few millinos of those, you can pay a fraction of a penny and still have a big black hole in your budget, unless you can significantly improve performance.

Only a few big companies can do it because there's a bootstrapping problem. To get large amounts of virtually free data, you need lots of users who have signed up for giving you their data in exchange for a useful service. This was much easier to achieve for companies started between 1995-2005, when the web was young, because the Internet was such a huge leap forwards over what came before it. Existing startups now have to compete with the products of these giants, many of which have been enhanced by years of machine learning. That's challenging.

To give you a sense of how cheap computing power is, my startup regularly processes roughly 2B webpages with some complicated algorithms that need to go node-by-node over the whole DOM tree. That's roughly 77TB of (gzipped) data, and around 100 trillion nodes. It costs me a few hundred bucks of AWS time. That's a rounding error for a big corp; a single data scientist's salary for one day will run you around that much.

> Logarithmic growth asymptotically decreases

Logarithmic growth slows down, but not asymptotically. Think about it: what would the asymptote be? (There is none.)

100 % accuracy is what the algorithm wants to achieve (in the most basic case), so that's your asymptote. If your precision increases logarithmically e.g. as 1-1/log(x) you get arbitrarily close to 1, which is the definition of an asymptote I think.

> 100 % accuracy is what the algorithm wants to achieve

Well, in that context every kind of (monotonic) growth is asymptotic, so the word has no meaning.

It really slows down though. Log(n) indeed goes to infinity but if you numbered all the atoms in the universe and pass them through the function, the last one would be ~80.

That may be true, but that is a far cry from asymptotic

Mathematically, yes. For all real-world intents and purposes, it's bounded. I sometimes use this very example to illustrate the profound difference between the two which can be easily under-estimated. (That or log(log(n)) which also goes to infinity but is real-world bounded at <3)

That is in log_10, in log_e it would be near 190.

Okay, yeah you make a good point. Strictly speaking a logarithmic function does not imply a convergent series. To be more precise, what I'm talking about are slowly varying functions. These are functions which don't converge but which have similar characteristics to convergent functions (namely, extremely slow rate of change).

Since you made the correction you're probably already familiar with these, but for the benefit of others: https://en.wikipedia.org/wiki/Slowly_varying_function

With diminishing returns it's interesting to think about this as a purely economic function when it makes sense to index more data.

If a 5% increase in accuracy worth more than the cost of data storage and computation then it's a pretty clear win.

I know it's an empirical result, but it would be interesting if it was truly logarithmic. Classic statistics gives various asymptotic results based on variations of the central limit theorem, which generally imply sqrt(n) scaling, but these assume various regularity conditions (typically smoothness and fixed dimensionality).

It would be interesting if there was some corresponding law for more flexible models which did indeed give you logarithmic scaling.

If the model is an ideal encoding, then the description length of every input grows logarithmically. Perhaps the performance is gated on the number of "features" in the ideal encoding.

Reply on news.ycombinator.com