I'd like to know the answer to this also, but after reading the comments I'm still clueless.
Maybe this will help - Has anybody dared to do any A/B testing on professional vs "unprofessional" design?
Maybe this will help - Has anybody dared to do any A/B testing on professional vs "unprofessional" design?
I know of a direct-response guy who had horribly ugly sales pages that literally included misspellings. He tells this story about how all kinds of marketing consultants and designers would come in and say, "Your sales page is so ugly! You need a redesign." He would laugh and say, "Everyone says that. But I have data from the last 5 years, and this is the single best-performing page of all." He made $25m last year.
Sophisticated direct marketers are not to be trifled with.
However, if you're building a company that depends on other non-direct factors like brand, you can't really A/B test that. You just have to make a decision.
A few years back, I met a guy whose girlfriend worked on the optimization team at Amazon. Now THEY are sophisticated. They knew how A/B tweaks would affect users several months later. But even still, Amazon often makes strategic decisions that can't always be justified with data.
Bottom line: It's important to know the advantages and disadvantages of various types of marketing, including direct/brand/etc. But you will have to make some tough choices.
Personally, on my sites, I've seen lots of interesting results with testing designs...but I finally told my staff to stop micro-testing since we'd optimized the hell out of some of our stuff and were only getting incremental gains (e.g., going from 35% opt-in rate to 39% is prohibitively difficult). The biggest results I've gotten have been from strategic changes like changing the offer, adding a new product, pricing changes, back-end offers, deeper customer research, etc.
In this age of analytics, I think a lot of small startups are overlooking the gut decisions they need to make and focusing too much on optimization (e.g. this HN question).
"Portfolio: Are you always extremely methodical about major decisions?
Bezos: With business decisions, yes. With personal decisions, I find that my methodical nature can confuse me, and so I think more about personal decisions, like what job you really want to take or whom you want to marry. Although I did have criteria for that...
Portfolio: What's a gut call you made?
Bezos: Amazon Prime. It's an all-you-can-eat buffet, $79, that gives you free two-day shipping on everything you buy for a year. When you do the math on that, it always tells you not to do it.
Portfolio: One of your big initiatives, a search engine called A9, fell flat. What happened?
Bezos: If you decide that you’re going to do only the things you know are going to work, you're going to leave a lot of opportunity on the table. Companies are rarely criticized for the things that they failed to try. But they are, many times, criticized for things they tried and failed at."
http://www.wired.com/techbiz/people/news/2008/05/portfolio_0...
Visual design as decoration is probably less effective, and I think you'll get a lot of that with "Unprofessional" designers. Visual design as a way to amplify/enhance existing well-designed interactions is definitely effective (i.e. strong button affordance, good use of color to convey meaning, etc). This is the type of output you'll get with a "Professional" designer.
Philosophy aside, I think you'd probably do better with qualitative testing. Do people tend to complete tasks easier if the action being tested "looks" easier? Does the brand identity match the audience you're targeting? Are your design patterns consistent across the site? Etc, etc. It's certainly a bit more nebulous but I think with the right questions you could certainly use A/B testing to answer some of these things.
This post talks a lot about this type of usability and speaks to this point (and even mentions A/B testing with usability): http://sixrevisions.com/usabilityaccessibility/10-usability-...
If the idea holds enough water, it doesn't NEED a good design (re: Craigslist, tarsnap, the original Digg, etc.) -- it just needs to work. If it works, and you can prove there's a business there, then by all means, spend the money to make it perform as well as you possibly can. Design in this sense is an 'optimization' technique, not part of the core product.
The flip side, is if you have a horrible idea, and you spend $10k on design, you've effectively thrown that money away.
The exceptions to this, of course, are in the demographic. If your target market is web designers, they're going to be hypercritical of any design, so it needs to excel from day 1. If your startup is an iPhone app, same thing. Otherwise, most people will be forgiving of a spartan design, so long as the damn thing works, and that'll give you leverage to design against.
Also, another big point in this is that as you build the product, you'll change how things work, a lot. In the application I'm building now, I had Photoshop mockups for every page, and how I thought they would look, but in the initial testing, I've changed almost every page somewhat dramatically. It costs me time, which I hate, but it's better than costing me time AND dollars I'd have to pay a professional designer to re-design because I changed the position of button x. And, while I hate having to re-implement, I'd much rather not move into production with button x in the wrong place because I couldn't afford to pay a designer to move it.