Hacker News Reader

Comment by Harj | Hacker News Reader

Marissa described possibly the most thorough and analytical job search process I've heard from anyone, when she was talking about how she joined Google. I really liked her reflection on this in hindsight on how being overly analytical is dangerous and it's something I try to remind myself of when I'm in danger of overthinking a decision:

"I think this is a common thing that very analytical people trip themselves up with. They look at things as if there’s a right answer and a wrong answer when, the truth is, there’s often just good choices, and maybe a great choice in there."

View on HN

Too much focus on utility functions, not enough focus on novelty functions, even though it's been proven that utility functions decline in usefulness as a search space expands. Given an infinite search space, a utility function can only find local optima, there is no global optima. In such situations, a novelty function that finds a path from one happy local optima to another happy local optima is a better bet than using a utility function.

The above paragraph is rational, and yet people who consider themselves hyper rational often ignore the truth of this. And the irony is that some of them do this for an emotional reason: they want the security that comes from believing that there is an absolute right answer. They are irrationally rational.

What an interesting perspective with a great paradoxical conclusion.

Well done.

How would you prove that?

Here's a simple counterexample to what I understand your theorem to say: consider an infinite search space: 𝕽∞ and a utility function: 1-|x|. There's a single global optimum at (0, ...), and the gradient of the utility function would find it quickly.

You missed the point that there is no global optimum. The GP’s novelty function is optimized for novelty, that’s all.

Counter example: R^1 with a random function. There is no algorithm that can find a global maximum other than checking every point in R^1, of which there is an uncountable number.

Not all cost surfaces are equally likely to occur in real problems... Also depends on the constraints, linear assignment (i.e. one job to one worker with a big matrix of cost for job to worker and you minimize the sum) has a polynomial complexity solution.

We are not talking about real problems here. Reality is so far from linear, so path dependent, so temporally dependent that by the time you gather 10 data points to try and match some function to the function is already outdated and error prone.

This is infinitely truer for when you try and find absolute maxima and minima and not just local ones.

Sure, some functions have no global maximum. But the comment I replied to claimed a theorem that every utility function on infinite search space has no global maximum, which isn't true.

All models are wrong, some are useful. GP presents an interesting way of framing real life decision making processes. That it happens to not be 100% accurate in all aspects is mostly trivia.

This feels like an opinion on how to live life dressed up in mathematical language.

>given an infinite search space

There's no such thing.

I understand the point you're making, but these gross assumptions aren't how the world works. Reminds me of econ models with ridiculous assumptions that don't pan out when reality is a constraint.

Doesn't matter. A 50 dimensional search space with 1000 possible values in each dimension has ~10^1700 possible states. That's a number you can't search exhaustively in the age of the universe even if you turned the whole thing into one computer. And this is not a large problem, you run into similar ones in the average gear wheel design.

The point of simplified models is to organize thoughts.

The only perfect model of reality is reality itself. Assumptions are fine if they're reasonable

> Assumptions are fine if they're reasonable

I would assume so...

ken?

Do you have some resources on novelty functions? Google, like usual recently, is useless at finding something I haven't found before.

This is a good book on the subject: "Why Greatness Cannot Be Planned: The Myth of the Objective" https://www.amazon.com/Why-Greatness-Cannot-Planned-Objectiv...

Much obliged, that book looks interesting.

This is an interesting line of thinking, certainly valueable in some sense.

It is also a very good example of exactly what the previous post refers to: overthinking stuff.

As someone who feels they're from the outside looking in (left college to work, still ended up in the technology but without a traditional college education) one of the most frustrating things is watch folks who I perceive as traditionally trained CS and similar folks ... is their desire to go hyper analytical ... and then REALLY commit to the result as the best choice above all others because of whatever analysis they made.

Now granted there are time to hunker down and commit but sometimes all that data doesn't really tell you anything and you're still facing an unknown no matter how much work you do, and it might be worth thinking about it after taking a few steps down that road / experience. It's not uncommon to come across a variable(s) that plays a far stronger role than any other, only AFTER you tried doing something.

For hyper analytical folks the data on hand is the hammer for every nail it seems sometimes.

I wonder if they recognize that luck plays a significant role. Right place, right time sometimes matters more than anything you can predict or control.

Life is simply a game with non-perfect knowledge. It's not chess or go, it's more like Dota or poker.

Even without unknowns people elevating rationality to something that would in consequence just be horrible for everyone. Shouldn't be too hard to see if you follow through with the consideration.

Hard data also suggest how often the allegedly rational result suddenly became wrong. The rational conclusion here should be to decrease hubris then, shouldn't it? Nope...

And this is already a stereotype for softies...

> "I think this is a common thing that very analytical people trip themselves up with. They look at things as if there’s a right answer and a wrong answer when, the truth is, there’s often just good choices, and maybe a great choice in there."

This absolutely drives me crazy in design/engineering decisions. Very commonly there are a lot of good solutions and one great one, and the good ones are good enough. Yet all the brilliant intellectuals want to find the VERY BEST METHOD EVER instead of just getting stuff done.

Do an anesthesiology residency. I love how, when residents with engineering backgrounds as undergrads run up against the immutable fact of 5 minutes of hypoxia = brain death, they quickly abandon their old way of thinking in which finding the optimal solution is paramount in favor of whatever works, however kludgy.

> Yet all the brilliant intellectuals want to find the VERY BEST METHOD EVER instead of just getting stuff done.

For many mathematical, CS problems, it _does_ help to think very hard to find the very best solution to the problem, sometimes irrationally hard. I do agree that we operate in a real world, and the facts of running a business mean that you can't be spending all your time trying to figure out the best.

However, it was only by thinking very, very deeply about these problems have many of the technological improvements been possible. MapReduce, AI, ML, Cloud Computing... all started as ideas in companies where people dedicate quite a bit of thought into how to solve some basic problems.

I'll be honest: I am glad that I can reap the fruits of the labor of all these smart people, that they have enabled me to change the way computing is done, to make it easier for anyone to get started and to generate value very quickly, using the building blocks which they created after thinking about and working about this for so long.

This used to drive me nuts too. Now I just look at it as an opportunity to outmaneuver folks who are too wedded to making their solution ‘perfect.’ (Whatever that means.)

Sheryl Sandberg described a similarly thorough weighing of her decision to join Google.

Mayer: "I had a long analytical evening with a friend of mine where we looked at all the job offers I had received. We created a giant matrix with one row per job offer and one column per value. We compared everything from the basics like cash and stock to where I'd be living, happiness factor, and trajectory factor—all of these different elements. And so we went to work analyzing this problem."

Sandberg: "After a while I had a few offers and I had to make a decision, so what did I do? I am MBA trained, so I made a spreadsheet. I listed my jobs in the columns and my criteria in the rows, and compared the companies and the missions and the roles."

It's a fun bit of trivia that Sandberg put the criteria in the rows, which enables sorting the criteria - a nice way to see the upsides and downsides of each choice.

I don't think this is a rare thing to do. I've done it and so have co-workers. A simple pro/con chart is really all you need.

Sorry to weigh in this way, but yes it's a rare thing to do. Most people don't have multiple offers sitting on the table. It's a privilege.

It's rare, that's all.

I'm referring to comparing your existing job to a new one. Everyone does this and it can't be rare.

You have to look at quality of work, work life balance, the area, commute time, cost of living, salary, 401k match, benefits, chance of advancement, company culture, job safety, bonus amount, job security...etc. For a lot of people the choice is a no-brainer, but there are comparable and even worse jobs out there.

I apologise for the seemingly privledged attitude, but I'm assuming most on HN are Software Developers, Engineers, Mathematicians, and Scientists which generally have options and change jobs on occasion. Every single person who changed has done a pro/con comparison. Even if it was a no-brainier, the comparison would've taken place subconsciously.

The issue is that almost all the factors you mentioned- quality of work, work-life balance, chance of advancement, company culture... - are all aspects you can only have the vaguest of ideas about before you start in a new company. And a mistaken evaluation of even a single one of them can change completely the score of the offer.

Not 100% true. It depends on the industry, but I have a pretty good idea of the good and bad of many of our competitors. You're right that some of those factors are fuzzy.

You shouldn’t apologise for a personal attack, whether on the internet or in person. Anyone who says you’re privileged is your enemy, at worst, and completely indifferent to your welfare at best.

> Most people don't have multiple offers sitting on the table.

Maybe not (although I don't think that's vanishingly rare).

But don't most people do something like this when deciding what they want to do next even if there aren't any active offers on the table at all?

I guess that’s more a factor of graduating from Stanford at that location at that point in time than it is of their personal ability to receive multiple offers (not completely unrelated of course).

"I had a long analytical evening with a friend of mine where we looked at all the job offers I had received. We created a giant matrix with one row per job offer and one column per value"

I think this is a luxury problem. How many people have competing job offers that are even close to each other in attractiveness?

Doesn't putting the criteria in rows disallow (easy) sorting? Spreadsheets are designed to sort by column.

Agreed. I think for most people here the risk is the indecision rather than the wrong decision. For a lot of people there's a deathly fear of ending up under a bridge, and while that's undoubtably true for many people unfortunately, I'd wager most people here have a lot more runway than they'd think even.

It's funny, this is the nugget of wisdom that stood out to me as well. I waste so much time in my daily life trying to make the "perfect" decision, when choosing something good and moving on would be a much better use of my time.

A good plan, violently executed now, is better than a perfect plan next week. - George S. Patton

I think about this quote a lot. It's so easy to get trapped in analysis paralysis which is really just procrastinating a decision. Like most things, there is a balance. Notice he says 'good' plan, not any plan.

The trick is training yourself to separate out the shit plans from the good plans. Otherwise, you’re violently crossing the Isonzo river for the 9th time and violently dying.

So, don’t be McClellan, but don’t be Cadorna either.

But that's an entirely analytical way of thinking about it! You're looking at the decision process and asking if the marginal return on investing another unit of time in it, in terms of the improvement of the goodness of the selected outcome, is greater than the return on using that unit of time in some other way.

The very term "overthinking" implies that there's a right amount of thinking for any decision, so your real problem is working out how much thinking to do.

Go for the girl or the boy.. as made famous in good will hunting

I've said something similar when mentoring engineering managers about how to let go of certain decision making. 90% of the decisions a team makes will have very little impact on the success of the project, but the other 10% do. You only learn from experience which decisions are the 90% and which decisions are the 10%.

It's how leaders need to operate to survive if they want to avoid micromanaging, honestly.

This reminds me of Jacob Falkovich's approach to picking a girlfriend:

https://putanumonit.com/2017/03/12/goddess-spreadsheet/

I'm going to reveal myself as entirely too geeky here, but my primary complaint about this approach is that it relies on a linear scale for evaluating utility, when much research suggests that utility curves are frequently logarithmic. (Example: Going from earning $20k to $100k per year is a huge difference with substantial implications for financial security, but $1m to $1.08m has a substantially lower impact.)

One could argue this article looks at a restricted range where the log behaves more lineary, but if we're going to apply mathematical modeling to our life choices, ... :-)

Yes, I agree - it was nice to hear her iterate that. I have learnt the exact same thing in my two decades as an adult: it doesn't matter really in the end what decision you make, it's how you make it work (and you do have to work at it).

Seem like one of those in group type biases. Where the more similar something is the more we obsesses over the differences. Presumably because we can relate to a lot more of the information.

Somewhat ironically being irrational can actually be a good way to make unknown, but largely equal, decisions. Because at least you picked something with conviction, rather than having analyzed the situation incorrectly.

Of course for a lot of us good choices aren't the problem so much as the downside. I remember someone made a calculator online for how many time one would most likely see their parents before they died.

Reply on news.ycombinator.com