Hacker News Reader

I feel like he's kind of invoking Goodhart's law that when a measure becomes a target, it ceases to be a good measure. Academia can only produce what's possible to produce under the constraints of meeting certain metrics. If you can't structure the work in such a way as to make the right metrics look good while doing it, you can't do the work at all, because you will be fired, and it doesn't matter how important the work is or why you need to do something that doesn't produce the desired metric to get it done.

Another way to put this is that the incentives in academia right now encourage "careerism", for lack of a better term. People have to put so much effort into maintaining their position that the actually big picture goals like research become difficult to do, especially when you have to show progress every quarter and funding is only given out in one to two year increments. Researchers end up playing it safe, because big risky ideas may pay off in five to ten years, but that kind of research is hard to fund on a yearly or even quarterly basis.

This. The other problem is that as academia becomes a careerist pressure cooker of competition, where the actual science takes a back burner and you’re incentivized to churn out high-likelihood-to-succeed but not-actually-profound science… why even go into academia?

Personally, I’d have loved to be a part of the Academia of the mid 20th century. But my exposure to research 10 years ago (thankfully, before committing to a PhD) quickly taught me that it’d be extremely unfulfilling, difficult, and poorly compensated compared to private industry. It would be cool to be a tenured professor I guess, but the coolness and prestige of that title is riding the coattails of the past when academia was less of a slog.

I get to work on novel CS problems (distributed systems) in my FAANG job without having to publish X papers per year while I progress on the actual meaty problems, for a lot more pay. It’s just strictly better IMO. I’d rather take 3 years from beginning to end to finish something new I’m actually proud of that’ll be used by hundred of millions of people, than be forced to publish papers very few people will read at a regular cadence or lose my job.

I completely reject Goodhart’s “law”. Good metrics are proxies for real things that cannot be gamed. If your metric does not have this quality, delete it.

I like the sentiment, but I'm legitimately scratching my head trying to think of any game-proof metric

Salespeople have fairly ungameable metrics - cash. That is the model we should be shooting for.

Consumer facing web products value depends clearly on number of users.

If a metric cannot be tied to something material, don’t start.

It's never that simple. Salespeople frequently want to do things that damage the brand. A lot of companies force them to present pitches a certain way to try to prevent that.

And you don't want 50 different sales people all trying to sell to the same client, undermining each other's efforts. That means you need some scheme to assign people to opportunities, such as sales territories. They will learn to game that system.

Sales are absolutely gameable and regularly are. Sales on false promises to customers, overpromising feature delivery, overstating financial benefits to the customer, are so common as to be cliche.

The customers end up being short term relationships, the sales provide negative value to the company due to reputation loss (not realized for perhaps years), but the salesperson collects the commission and is probably working somewhere else when the smoke clears.

And I'm not sure if you're claiming "number of users" is ungameable. If so, I'd invite you to reconsider that - we've just lived through an entire era where companies gamed user metrics to harvest sweet, sweet VC dollars. Founders game the metrics for VCs, vice presidents game the metrics for founders, product managers game the metrics for VPs.

If cash is gameable, then play the game. If I hire a machine to generate cash and they are consistently able to do and grow that it seems that this is a good game to play.

You could also reformulate the objective function to NPV.

The point is that not all cash is the same, and some immediate cash causes medium or long term harm. But if cash is your metric, it will be gamed by not distinguishing between healthy and unhealthy cash.

So you modify your metric to try to eliminate the harmful kind. But now you're in the same race as other metrics.

If NPV isn’t sufficient then you are essentially in the realm of maximizing happiness as the objective function.

You're begging the question by assuming we can tell what sales are positive and which are negative value. In other words you're asserting we have perfect knowledge and can't be gamed, to argue the measurement can't be gamed.

Here is a thought, introduce a new term "metric NPV correlation factor". If a particular metric has a perfect correlation with NPV it is assigned 1.0. If increasing or decreasing the metric has no impact it is assigned 0.0. Now that we have established a gradient, it is possible to further the discussion.

You are stating that cash / sales do not have exactly 1.0 but can be something less. I can accept that. However, the correlation is very high and higher correlation factors are harder to game. Metrics with high correlation factors also do not need to be hidden.

Let's contrast this with another metric: number of emails sent per month. Low correlation factor, easy to game, likely to actually cause harm to a business. This is the kind of metric that you would certainly need to invoke Goodhart's law on (i.e. hide) which is, in turn, a sign that a metric should not be used.

My main thesis is metrics must have high NPV correlation factors or should not used at all. High levels of scrutiny should be placed on those introducing such metrics. Finally, no metric should ever be hidden as it is a clear sign that NPV correlation is low by definition.

You've never heard of someone tanking the consumer's opinion of a company in exchange for some short term profit?

If you make the metric "cash over the past 20 years" that would be less able to be gamed, but do you really want to wait 2 decades between each performance review for your sales team?

There's also the problem of one person gaming the cash metric to the detriment of others. Like a used car salesman who lets other salesmen "warm up" the customer but then swoops in to be the one to actually make the sale. Is he really 10x better then the other sales staff or is he jut a leech claiming other people's sales effort as his own?

Even in the best case, where you create a flawless metric that cannot be gamed, it's typically just one aspect of the job.

If people get rated on that metric, they'll maximize it at the expense of the things that can't be measured.

But, there are tons of subtle ways to undermine metrics. Want to be the top salesperson each month? Give negative feedback to candidates for open positions that seem too strong.

Yes, definitely a case of Goodhearting. One way of further breaking this down is the idea of “slicing the bologna” where if you have an idea, you slice it as thin as you can into as many papers as possible. This you can cite your own previous papers, get more totoal citations & papers, and generally game the impact factor.

Additionally the “publish or perish” regime encourages small incremental research projects, rather than large expensive risky ones that might not produce a paper for many years.

All this against the backdrop that it is getting increasingly harder to find new discoveries since the low-hanging fruit is mostly harvested.

Lol... "slicing the bologna" is such a great phrase to describe this.

I've seen it happen a lot in both experimental and theoretical fields. Rather than publish one big paper that explains a new idea clearly at a high level + show its numerous applications, they publish many (6-10) small results based on the the "machinery" behind this idea. It can be frustrating as a beginner in such fields, like learning from a textbook that shows you just solutions to problems and doesn't explain the general theory.

:) I first heard it in sprinting, where they try to beat the previous world record by the smallest possible margin so they can claim sponsorship bonus from beating it next year too.

If the low hanging fruit is gone and producing profound results requires more effort you'd expect the number of papers to decrease, or at least it wouldn't be appropriate to use number of papers as a metric because producing good papers requires exceedingly large amounts of efforts and possibly luck.

While that seems reasonable, it assumes the amount of impact per paper has remained constant, while is practice it has greatly declined. These days scientists will literally try to identify the set of "least publishable units" in a research project to maximize the number of papers (I've attended meetings where this is discussed without irony). There is also a tendency to publish work that is quite similar to previous work with a minor change that is emphasized disproportionately to its importance.

Not if the scientists involved are p-hacking. Or, less harmfully, publishing null results.

Publishing null results is the opposite of harmful

Surely you mean _not_ publishing null results? From what I understand, null results are just as scientifically important to publish as alternative results, they just aren't published as often because they aren't as exciting.

Yes, I would expect amount of work per paper to stay roughly constant in a healthy research community. The whole point of writing papers should be to communicate what you did and found so that others don't have to do the same thing, and I think the amount of work would be the relevant variable that determines the amount of information that has to be transferred.

Part of it is also the fields are much larger. Both internationally of course, but also on a national level. Almost 150 R1 schools in the US now. More grant funded work is being done than ever before. More agencies funding research both public and private as well. More papers than ever submitted to more journals than ever by more people than ever, year after year.

Doing science and publishing papers is extremely different than pottery, and that has nothing to do with a difference in status between the two.

When doing any form of art, the most important thing is to practice. If you make 20 vases, the 20th is most likely far better than the 1st. Same with painting, drawing, other forms of sculpture... nothing teaches like practice. There's such thing as more complicated pieces that require more work, but the majority of actual skill at pottery is practiced the same way whether you are aiming for a relatively complex piece, or one that is just hard enough it doesn't bore you.

On the other hand, in science, there's no such thing as random practice making you better. The more papers you publish, the higher percentage of the time is dedicated to the writing bits, and getting through the approvals, and doing a million submissions, instead of coming up with good hypothesis, and finding something to do. Aiming at a hard problem is going to lead you to publishing less, not more, so the things you'll work on are going to be minor improvements, nothing difficult. Therefore, the chances that you'll discover anything major definitely decrease the more often you publish.

So if in your intuition you'd be better off focusing on making a lot of unimportant papers... why would anyone aim for any fundamental problem, which has a higher chance of failure? You think the efforts to actually publish and submit, all away from the lab aren't mostly wasteful? Because it's really hard for me to imagine how publishing more doesn't mean far less time doing research.

Interesting analogy with pottery. I see that an important academic practice that does scale like vases is structuring your research in easily publishable way. As a beginner (typically a young highly motivated idealistic student — who else would want to go to academia in the first place?), you can start with good research idea and important results and no understanding of academic world, but struggle to get accepted anywhere decent because you didn't follow all unwritten rules, and as you get experience, you learn the rules of the game and plan your work to contain many low-risk "least publishable units" to build your reputation. So yes, "a lot of unimportant papers".

Many interesting papers nowadays come from commercial R&D departments, like MS Research, because unimportant papers probably don't move your main KPIs (which would be "a lot of patents" AFAIU), and only interesting results improve the image of the company and serve as good marketing material.

>I'm also reminded of the classic story of the pottery teacher who graded one class by final project quality and another by total weight produced over the class period, and found that the latter group ended up producing higher quality work. Higgs in this article seems to be of the persuasion that in his profession, the former style produces better results. I think many of us here would find that intuitively wrong in general.

Wait, how is pottery equivalent to significant expansion of the bounds of scientific knowledge? Humans mastered the contours of pottery thousands of years before farming was a thing, we have no such bearing wrt scientific knowledge.