Results from Candy Japan box design A/B test | Hacker News Reader

Results from Candy Japan box design A/B test

candyjapan.com

121 pointsNickSharp93 comments

In a past life I founded/operated a conversion rate optimization company.

One of our all time most successful tests was related to this story and happened almost by accident.

The client had just rolled out new packaging and wanted help validating the change statistically, very similar to this situation.

Their metric was measuring upsells to a premium product after a purchase, and like Candy Japan, found statistically insignificant results.

But the new packaging was beautiful and we wanted to test it on their landing page.

We tried just replacing the old product image on landing page. Meh results, no real bump.

We tried increasing the size of the image + circling with "Improved" (no joke). 10% bump. Interesting.

We hired a professional model to hold the product and smile. 15% bump. !?

We hired a group of professional models to hold the product and smile. 40% bump in sales. !@%K$!?

We re-hired the same group of people to hold product, look SUPER excited (jumping up and down like their team just won the superbowl). 50%+ bump in sales.

No joke.

If Candy Japan is reading and you'd like some help recreating I would not be shocked to see something similarly group oriented, visceral, and beautiful bump yeh.

I wonder if you could have gotten a similar result from the same models holding the old box?

Alternatively, what if you already had the models and wanted to improve?

this is the million dollar question

Interesting story! I've long wondered if the (over)excited voice of many a startup's marketing/onboarding/help copy is justified by customers liking it more than the alternatives. This seems connected.

>10% bump.

>15% bump.

>40% bump in sales.

>50%+ bump in sales.

Was that cumulative (i.e., 1.1 * 1.15 * 1.4 * 1.5 ≈ 2.7 total) or relative to the base figure?

Advertising is important to set a level of trust in the product

How many sales would Amazon have today if they went back to their 2000s website look?

The decorated box might not even have the best drawing, but indicates that the salespeople went the extra mile

Hmm, let's see. Beautiful Japanese woman(en) in form fitting outfits holding product, smiling. 940% bump I'd bet.

How about a plain box in some relevant color (sakura blossom pink? :) ), hand-stamped with the Candy Japan logo?

In my mind's eye, that'd remain classy enough not to be creepy/cringy/embarrassing (as https://news.ycombinator.com/item?id=11641715 complains), but would also have sort of a personal touch to it.

Having a couple rubber stamps made and stamping each box during packaging wouldn't probably cost an awful lot...

It would probably be cheaper AND would feel more "authentic".

After all, his main edge over the local japanese stores is authenticity. The fact that it comes "directly" from Japan.

Seems like a good alternative. There has been a lot of valuable feedback in these comments. I hadn't considered how these might be perceived at a workplace, or that people even receive packages at work.

I second having something more classy. Japan is renowned for its package design, so why not include this aspect as part of what's being sold? Maybe something like how Meiji brands their goods. That illustration/design would seem to me to make you less differentiated from your competitors who are tacking on candy to their other "nerd culture" offerings.

Have you considered that the new box is too "generic" or "corporate?" (For lack of a better word)

In other words, the old box, while plain, adds a certain intimacy. It doesn't look like a huge multinational candy company sent it. The new box, though, looks basically like any other piece of Japanese media.

I can see how that might be the case. Current subscribers might know me personally or through my blog. That would then mean that as I start to reach out to people who don't know me personally, these results could change.

Unrelated: the jpg images in this post are huge in both dimension and filesize. Might be better to use smaller versions that link to the full image.

As a current subscriber (who has been receiving the old packaging), the new packaging is off-putting:

* It looks childish. If you are familiar with the Japanese style of advertising etc., probably not an issue, but as a box to be sent anywhere and to be received in a variety of settings (workplaces etc.) it seems like a poor choice.

* Anime may have poor connotations (unfair IMO) as pointed out elsewhere in the thread, which will reflect on the receiver.

* It feels corporate / like advertising. It may seem silly, but I'm more likely to continue subscribing to a small company if I'm on the fence about canceling.

* It feels like a waste of money. The current packaging is utilitarian, which signals that money is being spent only where it needs to be. The new packaging feels like an unnecessary indulgence, which would make me think I'm getting less value out of the subscription.

For me, the plain box gives me cheap Chinese vibes. It brings to mind badly damaged cardboard boxes with dubious electronics inside.

The Japanese illustrated box is innocent and kind of cute (certainly not "rapey" as another commenter put it) and in a Western country, stands out from the things around it. Perhaps if you're in Japan, surrounded by Japanese media it might seem generic but I don't think that's a problem elsewhere.

In addition, Japanese products are not unknown for overpackaging. It's probably not surprising that Apple took some cues in their packaging from Japanese product packaging to signal "upscale" meticulousness and attention to the customer.

I often think like this, then I'm reminded how infomercials are so obnoxious and yet they still work. So the conundrum of personal taste and a good gut instinct vs what works with a lot of people.

was the old box that blue plain box? It's not intimate to me. Compared to the new one, it looks kind of homely

I much prefer the plain packaging because:

- The anime picture is embarrassing to receive in the mail / at work.

- The picture itself is kind of creepy, someone walking past my desk once even described it as 'rapey'.

- I can reuse the box.

- I am not 6.

"Rapey"? Are you serious? I'm just really having trouble wrapping my head around this. How does that picture convey anything close to "rapey"?

Um... perhaps by sexualising a young girl?

In Japan you have 30 year olds (and older) who dress in that manner[1] and behave in that manner and speak in that manner. It's just different --so viewing it under a western lens probably introduces misinterpretation.

[1]Just look into Rorita fasshon, etc.

I believe they're talking about the art on the box, not the picture of the cosplay girl holding it.

Honest question - what do you see as sexualising in that picture? (I'm not going to question the young girl part, but I think that depends on culture / drawing style / reception)

I believe that's a picture of two happy people sharing a box of candy. I think you're looking a bit too deep into this one.

Two (seemingly young) girls, one of who is winking at the recipient? Plus this: http://www.urbandictionary.com/define.php?term=candy

I never got a rapey vibe from my grandmother when she winked at me while sneaking me a bit of candy out of my mother's eyeline.

Maybe I'm the weird one here?

>Plus this: http://www.urbandictionary.com/define.php?term=candy

It's a box of candy. It's not a metaphor for sex, it literally contains candy.

I don't agree with the connotation, but unfortunately there are negative associations in American culture with older white men who are into anime or other Japanese cultural artifacts.

Sure, there's a certain stigma attached to anyone who enjoys Japanese culture -- older white men included. But even then, one of the words that pop into my mind when thinking about said stigma isn't rape. It's weeaboo, nerd, otaku, geek, parents basement, neckbeard, etc etc etc.... I say that as someone who enjoys japanese culture from time to time myself. Not anime, but j-dramas, movies and their music. Especially their music. It reminds me of how music was here in the 90s (not as manufactured, still a lot of experimenting going on, etc).

Those are great points. I'm actually hoping to be convinced that I should stop using the new box, because it costs more.

Maybe I should start calling this the plain vs. rapey test.

Yeah, pretty sure if customs kept seeing boxes like that appearing they're going to start investigating people...

…for what?

Why not leave it as an user configurable option. I personally think that the new box looks better.

> - The picture itself is kind of creepy, someone walking past my desk once even described it as 'rapey'.

This person may have some deeper issues than just the innocent picture on this box.

If we replaced all pictures of females on candy boxes with males I would hazard a guess that someone would still have something to say about it.

Well put.

It depends on the audience.

Someone who admires Japanese culture will likely not have the same reaction as someone with little exposure to Japanese culture.

The picture is not even on the same planet as 'rapey' Whoever that was either said it to get a rise or they have some distinct issues.

One solution, perhaps, is avail both packaging options to buyers.

The typeface and font, I think, could be improved. It kind of looks like an afterthought. I don't see "branding". It's not Lobster or Comic Sans... but...

> It depends on the audience.

One problem is that even if the person receiving the box is exposed to Japanese culture, their coworkers probably aren't.

So I think the main issue is how the buyer might perceive the packaging. If a coworker has issues... I think the coworker has issues. It's like someone finding Barbie offensive (for other than body image issues). I can't see it. Are these people who whisper and shuffle when they buy condoms or feminine products at the checkout counter? Do they wrap their wine or beer in brown bags? It's hard to imagine this person.

Humans are social animals. When I buy things, I absolutely consider how that purchase will look to other people. it's one of the primary reasons why brands have value.

You might not like that the world works that way, but as a business it's wise to not deny reality.

Right, but this nettlesome coworker you imagine is, I can only imagine, the lone outsider who can possibly find offense in the packaging. I think most would be like, hey, what's that? Oh, candy from _Japan_? Can I try one> Wow, Oishi!

I can guarantee you that nettlesome coworker is not alone in perceiving creepy links between Japanese cartoon characters and hentai.

That's the kind of person you do not want to work with. This the conspiracy theorist who misinterprets benign things and paints them in whatever twisted way they want for whatever odd reason they have.

_That_ person is the problem in the office, not the person who bought sweets online and probably shared them with some of the office.

If anything, the packaging helped reveal an oddball at the office.

> I absolutely consider how that purchase will look to other people. it's one of the primary reasons why brands have value.

Wait, what? In fashion, sure, by definition - but when I'm choosing between Persil and Tide, or Coke and Pepsi, or Pizza Hut and Domino's, I'm primarily looking at how good it is at whatever I want it to do. Which one cleans better, which one tastes better?

There's a certain point where many people start buying things more as social signalling than as actual useful things, but I'd disagree that the vast majority of brands could be fashion items in this way. The rest use their brands to evoke trustworthiness - "I know that this will do what I want".

Masking one's preferences and behavior to avoid offense in a public is a defining element of Japanese society and culture. And while everyone there is more inured to these kinds of graphics, the design of packaging for everyday goods often veers towards the minimal/elegant for precisely this reason.

https://en.m.wikipedia.org/wiki/Honne_and_tatemae

That presumes this graphic is offensive which I feel it's not.

That said, that does not explain people reading the kinds of graphic novels and magazines they do in plain sight in public on trains in parks, etc.

I hadn't intended to imply that the graphic was offensive, nor do I think that most Japanese people would find it so either. I was just pointing out (perhaps imprecisely) that perceived appropriateness in Japan is highly context dependent, with work environments being amongst the most conservative in this regard. The degree to which people seek to avoid offending others (as compared to the US) is quite at odds with a "if someone doesn't like it, it's their problem" mentality.

Trains and parks, by contrast, are generally considered an appropriate context to consume these kinds of materials. That being said, book covers are also far more commonly used for exactly the same reasons.

Yeah that's something I find really embarrassing - so much so that my first thought was 'omg I have to cancel this service' when I saw it

Eh, I'm pretty sure most people in Japan would mutter 'hentai' if they saw that box on your desk. It's childish (at best), even by local standards.

That has to be a joke. This has no hint of tentacles or sexualization that I see. It's a couple of youngish* people sharing or (long shot, if you take the greedy view) tugging over the box of sweets.

*The age is hard to tell, it's like trying to divine Buggs Bunny's age from his cartoons. But the age is irrelevant as it's two people with a box of sweets.

I believe in Japanese "hentai" just means pervert, or perverted. It's only in the West that it's associated specifically with tentacle porn.

You're still right, though.

>Eh, I'm pretty sure most people in Japan would mutter 'hentai' if they saw that box on your desk.

You mean "kawaii," right?

Maybe if it was teenage girls all the way around. If a 25+ year old male had that laying around, I guarantee you all the women I know would non-judgmentally classify him as 'probably hentai'.

Yeah I think it's really borderline, besides the product is candy - not girls so why make girls the main / standout part of the very publicly visible packaging. It just looks... Dodgy!

Sort of... It follows Japanese advertising characteristics and they are selling Japanese candy. So.... But, yes, I think the people depicted are too prominent and the candy should be more conspicuous. The packaging might benefit by giving the candy more prominence?

I had to a do a double take on the logo typeface after I read your comment. Turns out it is Lobster.

Rapey? A cartoon person giving a box of candy to another cartoon person, no touching, both fully clothed, both smilingly happy?

I think you're projecting your own issues quite considerably.

The "someone" who described that picture as "rapey" has some deep issues.

Perhaps, but it's not a huge jump from creepy given how hyper sexualised girls are in modern society coupled with perhaps some good intentioned coworker humour but at the end of the day I'm paying for the interesting candy, not for cartoons of young girls and I find that weird regardless.

I run a subscription business through mail myself, and I love your posts.

Have you tried to run introduction offers? 50% off for first shipment for example? If I were you I would try to put the countdown for the next shipment on top of your site. Thats a really powerful psychological factor.

Your CTA is also a little unclear, why do I need to first get an email, then a link? The buy process should be straight forward, not through my email. And if you have a really good reason for doing it this way, you should atleast tell me why I need to leave my email.

I really like the cartoon that explains the process. Very good.

Regarding packaging design, we did a huge remake in 2013. We increased our prices with 30%, and redesigned packaging completely. Doubled our LTV, insane gains. Just keep testing, but try to change provider for carton. We pay $0,1 USD for bigger cartons then yours, in one of the most expensive countries in the world. (With plastic window for address field) We chose to put our three USP on the back of the shipment box, to remind your customers every time they receive a shipment why they subscribed in the first place.

Keep testing!

* Could you give me more information on how you sourced your cartons?

* What does LTV mean?

* What does USP mean?

LTV = Lifetime Value = How much money you make off your subscriber over the lifetime of their subscription

USP = Unique Selling Point = Why your subscription gives the subscriber something they can't get anywhere else

I don't really think A/B testing something like this, with this size of a sample, on a subscription based niche-product can give you any real insights.

Why not poll your subscribers or (assuming you can get the costs to work out) have an option for anime art. Personally I wouldn't want the anime style box, something custom and simpler would be fine.. but thats my personal opinion.

> In a world where it makes no difference which package is sent, you would get a result as significant as this 80% of the time.

I suddenly get p-values now. Why has no one ever explained it like this before?

Thank Marek Gibney of http://www.productchart.com, he contributed that line.

I agree, I'm going to start using it unless someone can tell me that there's something wrong with or it is prone to misinterpretation.

Just a small update, but I just ran a survey for customers to see which package they would rather have.

Which package design would you rather receive?

The white one with illustration 79%

The plain dark one 21%

Total votes: 52

I'd suspect much different results if you conducted the same survey but indicated that the white box may result in cost increases to the customer. I suspect they wouldn't care about the box if it meant a higher cost.

I didn't get in on the survey and got the old box in the A/B test, but I would 100% prefer the white one with the illustration for my subscription. I order my candy to the office to share and people usually see the box from Japan and are curious so they ask me about it and I tell them about the candy subscription.

The white box would be better advertising for you in my case :)

I like my subscription. My biggest issue has been shipments that end up in my mailbox melted. It would probably not be possible for new packaging to fix this, but this would be an area of improvement worth higher cost.

a light-colored Vs a dark colored box might help in that regard as well with exposure to sunlight.

I think candyjapan should add a new form of subscription, one-time shipment, and I believe they would see it being ordered by about 40% of their customers. ;)

One consideration is the user experience is not connectable to a purchase.

If I went to a friends house and saw this, I would never remember the name. If you could combine "making the purchaser look cool" with timing-relevancy, I think your packaging improvement would pay dividends.

Put another way, I believe your A/B ratio test is actually X*Y/Z where X/Z and Y/Z are both < 1. Synergy.

Terrible idea, (possibly) useful for analogy: Instead of optimizing the unboxing experience alone, if you also included cards with QR codes, where every QR code immediately triggers a free candy bar (with user data entry on a mobile app). It should also provide social value to the host in realtime; maybe they also receive an immediate notification that their guest scanned their card plus some other bonus I can't think of.

The point is to get closer to the conversion/churn decision, or provide more value to your consumer.

I signed up for CandyJapan several years ago when they were just getting things started. I canceled after probably 3 shipments. Each shipment I was only hoping that the next would contain better products and actually be worth the value in which I was paying. Unfortunately that never happened. While a cool concept and very eye catching, that's all it is. Importing candy is expensive no matter how you go about it. Most people just want a poke to see how it is, and then move on to something else.

Thanks. Yes it costs much more to send candy in individual boxes from here, so if you want more candy per dollar it is definitely better to visit a Japanese market if there is one near you.

Both packages look pretty ugly. Hire a better designer...

Could the customs sticker have caused problems?

I used to receive Candy Japan in the "plain" box to Australia, and the customs sticker declaring the value & contents took up most of the space on the box.

Just a FYI: The pictures stopped working here: https://www.candyjapan.com/mini-assort

Had those pictures in the app engine blobstore, not sure what happened.

Why not allow the customer to choose the box type and default to the plain box. Add $1 for the designer box, see what people choose?

Now you have to try out different designs to see if it matters (it does and it can).

Just out of curiosity, which statistical test should be used in this case?

Some decent answers on Quora:

https://www.quora.com/What-statistical-test-should-be-used-f...

Those answers are bizarre. Use a t-test or z-test?! That is both less interpretable* and less efficient than doing a test of difference in proportions (based on a chi-squared) or better yet, doing a generalized linear regression with a binomial link which lets you test multiple interventions and include covariates and other things. These are little harder than using a t-test and work much better.

(Of course, even better would be a Bayesian decision analysis which takes into account the full posterior distribution, costs, and benefits, allowing one to decide whether the evidence is enough to justify the more expensive box or whether one should extend the experiment further to collect more data.)

* quick, if conversions are 0.25%, what does a 0.1SD mean difference between arms of an A/B test mean? 'Uh...' Exactly.

To be fair, a couple answers there mentioned the Chi-squared approach, and a z-test could also be appropriate for testing a difference in proportions. I think your Bayesian approach would make the most sense for this kind of stuff, to examine the posterior especially. There's other things that could be done than what you mentioned too to assess a difference. I'm not convinced the parametric assumptions made by the standard tests mentioned (including difference in proportions, whether a z-test or Chi-square test) would hold up well for this kind of data, it would take further examination. I think a z-test or Chi-square test are a decent starting point, and could be fine to use if the assumptions made for those tests are met (to defend some of the Quora answers). The issue is often tests are used without folks either knowing how to check or ignoring the underlying assumptions, but that isn't an issue with the test itself I'd argue.

> I'm not convinced the parametric assumptions made by the standard tests mentioned (including difference in proportions, whether a z-test or Chi-square test) would hold up well for this kind of data, it would take further examination.

I'm not sure what you mean. If all you have is binary data, there's not much you can do but a chi-squared test or binomial regression. As I understand it, a chi-squared is considered nonparametric: https://en.wikipedia.org/wiki/Chi-squared_test https://en.wikipedia.org/wiki/Fisher's_exact_test

Yes, you're right that chi-squared is non-parametric, my mistake. However, even non-parametric tests can have assumptions that should probably be checked into at least for a given data set. It may be robust to the violations the data collected may exhibit, I'm not sure. This article lists the assumptions for chi-squared test: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900058/

I would suggest a Bayesian decision-theory test of mean profitability: http://www.gwern.net/Candy%20Japan (Negative, so the decision is to use the old box.)

That 30 day shipping period is a killer. Candy is an impulsive treat. :)

Honestly, I think it would have been really bizarre if something as unrelated to the central product as the design on the box, changed something as fundamental as retention of customers.

That works under the assumption that humans are purely rational beings, which we are not. Though I don't necessarily think the design chosen was a great choice based on other comments here. Maybe something a bit more generic, yet still colourful (like the candies in games like candy crush) would have been a better choice.

Maybe, and if they were selling something for display, like figurines, or even a quasi-collectible I would be right there with you. I know people who like that kind of thing, and they would like a funky box too. This is JUST candy though, it's just people buying something they want to eat. It's a different mindset, and I think, less likely to be engaged by the aesthetics of the packaging (as in the box). They're interested in the packaging of the actual candy, and the candy itself.

All of this is "I think" and "IMO" of course.

Reply on news.ycombinator.com