Comment by kevlar1818 | Hacker News Reader

Comment by kevlar1818 | Hacker News Reader

Okay, this is awesome (and easy to miss with a just a cursory skim):

"5.3 Delicious Chocolate Chip Cookies

Vizier is also used to solve complex black–box optimization problems arising from physical design or logistical problems. Here we present an example that highlights some additional capabilities of the system: finding the most delicious chocolate chip cookie recipe from a parameterized space of recipes... We provided recipes to contractors responsible for providing desserts for Google employees. The head chefs among the contractors were given discretion to alter parameters if (and only if) they strongly believed it to be necessary, but would carefully note what alterations were made. The cookies were baked, and distributed to the cafes for taste–testing. Cafe goers tasted the cookies and provided feedback via a survey. Survey results were aggregated and the results were sent back to Vizier. The “machine learning cookies” were provided about twice a week over several weeks...

The cookies improved significantly over time; later rounds were extremely well-rated and, in the authors’ opinions, delicious."

I worked for Google at the time they went through the cookie iterations in the cafes. Personally I disliked all the cookies that ever came out of this project, mainly because they contained spices, I heard the same thing from many others.

After you got handed a cookie you were asked to fill out a feedback form on a tablet, which almost always is annoying.

But more importantly, the process of optimizing the ingredients left out many parts of what makes food enjoyable, temperature and texture for example.

I know this was meant to be a fun showcase of ML but to me this is still my favourite example of explaining misuse of ML technology where a simpler statistical model supported by expert opinion would have outperformed the taste of the cookies. Eg, any cookie expert might know that the number of people who like spicy cookies is only a small subset of all cookie lovers.

If I recall, the spices were an intentional choice of a kind of pathological input to the optimizer, and iirc it did what you might expect (minimize the amount of cayenne to the smallest in the acceptable range).

Sounds like you got the less tasty side of the explore/exploit tradeoff.

I think this is exactly the case how ML could go wrong. I would assume the people who hate the cookies probably will not bother to submit a survey and try again. The sample might be biased and not representative of the general population, thus overfitting, performs badly on a boarder test set.

Our dystopian future where we will be forced fed cookies in order for accurate optimization.

There was an icecream advertisment with that as a theme.

https://www.youtube.com/watch?v=j4IFNKYmLa8

That already happens :-)

I've heard this was a problem since the spice level was optimized in an office where they make the cookie dough the day of service, but deployed in an office where they make the dough the night before and cook on demand. The spices in the dough longer made the flavor stronger than it was optimized for.

A good example of training/serving skew.

I wonder if there was an ethnic bias at play with the population that participated since spice level tolerance is higher in Asian countries.

https://diversity.google/annual-report/

It's great that they published this - I wasn't sure if I could share this externally. I see they left the ingredient that really improved the scores unexpectedly out, presumbly for proprietary reasons.

Weed?

You're not thinking of these gluten-free cardamom cookies? https://www.blog.google/technology/research/makings-smart-co...

That ingredient list is missing the "secret ingredient" that your parent post is talking about, so nah.

They most likely are holding it back for a bigger media event where they will show them off and every tech blog on the internet will run a story about them.

Are you talking about orange extract? or cayenne pepper? The post is here: https://ai.google/research/pubs/pub46507 and direct link to paper is here: https://storage.googleapis.com/pub-tools-public-publication-...

And the recipe I found internally is "The Real California Cookie" in that paper:

Bake at 163C (325◦ F) until brown:

• 167 grams of all-purpose flour.

• 245 grams of milk chocolate chips.

• 0.60 tsp. baking soda.

• 0.50 tsp. salt.

• 0.125 tsp. cayenne pepper.

• 127 grams of sugar (31% medium brown, 69% white).

• 25.7 grams of egg.

• 81.3 grams of butter.

• 0.12 tsp. orange extract.

• 0.75 tsp. vanilla extract.

What an odd combination of units. Here's a translation:

• 5¼ oz. of all-purpose flour.

• 7 oz. of milk chocolate chips.

• scant ½ + ⅛ tsp. baking soda.

• ½ tsp. salt.

• ⅛ tsp. cayenne pepper.

• 4½ oz. of sugar (31% medium brown, 69% white).

• 1 oz. of egg.

• 2¾ oz. of butter.

• ⅛ tsp. orange extract.

• ¾ tsp. vanilla extract.

Or, equivalently:

• 167 grams of all-purpose flour.

• 245 grams of milk chocolate chips.

• 2.96 mL baking soda.

• 2.46 mL salt.

• 0.61 mL cayenne pepper.

• 127 grams of sugar (31% medium brown, 69% white).

• 25.7 grams of egg.

• 81.3 grams of butter.

• 0.59 mL orange extract.

• 3.7 mL vanilla extract.

Still no good because what the hell is 1oz of egg?

iirc, if avg = 50g, yolk is 20g, white is 30g... still up to the foodmage to interpret which part for 1oz (yolk adds taste/color, white has binding properties).

Meanwhile I'm feeling pretty good for buying both the 1/10th tsp and the 1/8th tsp scoops right now.

Unfortunately my microliter precision pipette is in the shop otherwise I would be enjoying some of these right now!

Can we get it to push out more useful output on how precise these measurements have to be to affect taste?

(say, as spectra)

An article linked in a sibling comment to yours mentioned szechuan pepper but it didn't show up in any of the recipes I've seen so far.

Good application for Ben Krasnow's Cookie Perfection machine - http://benkrasnow.blogspot.com/2014/01/cookie-perfection-mac...

Seems to be missing the key ingredient of the best tasting chocolate chip cookies: espresso powder.

Is this possibly related to the cookie machine by Ben Krasnow from several years ago? [1]

[1] https://www.youtube.com/watch?v=8YEdHjGMeho

Did they post the recipe?

They didn't.

I am very disappointed. I, for one, welcome our new robot overlords and also want to try their chocolate chip cookies.

Here you go:

https://www.blog.google/technology/research/makings-smart-co...

These are not the vizier cookies.

Oh darn you're right (to clarify, I'm pretty sure those are vizier cookies, just not the vizier cookies). I have the recipe sitting on my counter at home. I thought it was public, but it doesn't' seem to exist online anywhere.

The secret formula is love!

https://research.google.com/pubs/archive/46507.pdf

I heard they also provided samples of the cookies along with the recipes/writeup at NIPS.

Yup, they did. They handed them out at their booth.

Why I have an idea about receipe optimisation as a service ?

I noted this section when I read the paper as well...

This is cute, but "in the authors' opinions, delicious" does not contribute anything to scientific research. Yes, you can use black-box optimization for cookie recipes. No, you should not make any sort of performance claims nor talk about "significant" improvement unless you back it up.

That's the authors' sense of humour. The more relevant part for science is "extremely well-rated".

Their back up: later rounds were extremely well-rated

How about "My brand new image recognition architecture performs extremely well on benchmarks"

But it's worse than the last 100 SOTA models...

> in the authors' opinions, delicious

Can there be/Is there an objective way to measure deliciousness? If not what can they really say?

To the extent that deliciousness is a physiologically measurable phenomenon, or there are proxy measures that correlated with it, yes, there is an objective way. You give large numbers of people cookies and have them rate it. Tastes vary but generally if you make something with umami, crunch, fat, and sugar, people will say it is delicious.

> Cafe goers tasted the cookies and provided feedback via a survey. Survey results were aggregated and the results were sent back to Vizier.

This, from the paper, is a great start on measuring deliciousness. But if you go to that effort, why not provide a benchmark? Such as the existing chocolate chip cookie recipe or the average scores from 10 random chocolate chip cookie recipes?

As a reader, I might suspect that performance was not that great compared to baselines but that the authors were able to mask it by making this qualitative claim ("the cookies were delicious") -- which I'm sure was still true!

This technique is known as anchoring and should be used alongside a descriptive scale when making subjective comparisons.

Yep. There's a lot of pre-existing science that this research ignored ...

But it really is not a very great start on measuring deliciousness.

Like said elsewhere in this thread, if after a few batches the algorithm optimizes into a local maximum gully that some people don't like, but others really love, then the ones that dislike it will stop taking the cookies and thus stop rating them.

But does that mean they're less delicious? Nobody knows! Because certainly the first and foremost great start on measuring deliciousness, is doing research on what is deliciousness, and are there any good methods to measure deliciousness that can be compared with other literature and research, and if that fails, either measure something else or defend why you chose a target metric that can't be measured properly. The latter being valuable if your way of measuring is novel and you think it's better than what has been used so far in the field. But you gotta provide a good argument for that, and the method used in this experiment wasn't particularly novel, simply flawed.

I mean you can be all giggly-googly about it, but food science is a thing.

This really assumes that there is one-true-deliciousness factor, which is patently false if the spaghetti sauce optimization exists. (in which the previously unknown chunky sauce actually was discovered to be a non-trivial market segment)

This seems more like a spanning/optimization problem (how to cover the majority of preferences while not needing too many varieties) than an actual optimization problem, but then again it's just a sample use case to use to sell their API, so they probably didn't think too hard about it.

Reply on news.ycombinator.com