2,126 karma · joined October 17, 2008
It is possible one day you will go to sleep biologically and wake up non-biologically. It’s just a matter of sufficiently emulating the processes that were present when you were the biological you.
Flattered to see this hit the front page! This is a project I’ve been passionate about for a while and been keeping it mostly under the radar.
Now that it’s public, I’m happy to answer any questions the HN community has.
Also, A/B testing is far from dead. Companies that have a culture of experimentation have grown their market cap by 756% in the last five years. See our Opticon keynote for more context: https://www.optimizely.com/opticon/
In case you want to learn more, I posted "Three Timeless Truths Entrepreneurs Can Learn from Alexander Hamilton" earlier today based on what I learned from the biography: https://news.ycombinator.com/item?id=11475047
Optimizely is a website optimization platform. We enable businesses to show the right thing to the right person at the right time. Our first product makes A/B Testing easy. In 5 years we’ve grown to become #1 in the category with 8,000+ paying customers including Starbucks, Disney, and Salesforce. We're profitable and our revenue is growing 400% year-over-year. Last May, we raised 58 million dollars to allow us to continue building great new products. Join us in our mission to empower businesses to make better data-driven decisions.
About the Job: We have scale: 7,000+ customers (including 100 of the top websites in the world); 120 billion server requests/month You’ll work on our amazing visual editor - a cutting edge tool that is the core of our product - and will enable it to do even more awesome things for our customers Work in small teams, in a continuous integration environment; we’re fast and nimble: 1-2 deployments every day
About Us: You’ll work on our amazing visual editor - a cutting edge tool that is the core of our product - and will enable it to do even more awesome things for our customers. We are also building a mobile SDK for both iOS and Android and are working on some great new products.
Requirements: A track record of being able to ship product, independently and/or as part of a small team. We will leave out the long list of languages because we would prefer a great engineer to someone who knows some Ruby.
Bonus Points: Passionate about front end technologies, you have a curiosity that motivates you to experiment and keep on top of technical trends Practitioner of progressive enhancement, and responsive UI design Good at debugging using dev consoles, firebug, firecookie, or any of your favorite tools Fearless - you’ll tackle a massive JavaScript codebase, and are willing to prototype crazy, awesome ideas
Perks: Free city-wide Gym membership at any Crunch Fitness. Free Clipper Card to pay for your commute from anywhere in the bay area. Catered in-office lunch and dinner on weekdays. Full medical insurance with very low co-pay and deductible. HMO, PPO, and HSA options available. Full dental coverage including orthodontics. Full vision coverage including contacts. Dependents 100% covered for medical, dental, and vision. Unlimited vacation policy. 401k benefit. Top-of-the-line MacBook Pro or MacBook Air and 30" monitor. Working with a great team and having a huge impact!
If you're interested, please shoot me an email at Ryan.Jordan@optimizely.com!
#1 - “Optimizely encourages you to stop the test as soon as it reaches ‘statistical significance.’” - This actually isn’t true. We recommend you calculate your sample size before you start your test using a statistical significance calculator and waiting until you reach that sample size before stopping your test. We wrote a detailed article about how long to run a test, here: https://help.optimizely.com/hc/en-us/articles/200133789-How-...
We also have a sample size calculator you can use, here: https://www.optimizely.com/resources/sample-size-calculator
#2 - Optimizely uses a one-tailed test, rather than a 2-tailed test. - This is a point the article makes and it came up in our customer community a few weeks ago. One of our statisticians wrote a detailed reply, and here’s the TL;DR:
- Optimizely actually uses two 1-tailed tests, not one.
- There is no mathematical difference between a 2-tailed test at 95% confidence and two 1-tailed tests at 97.5% confidence.
- There is a difference in the way you describe error, and we believe we define error in a way that is most natural within the context of A/B testing.
- You can achieve the same result as a 2-tailed test at 95% confidence in Optimizely by requiring the Chance to Beat Baseline to exceed 97.5%.
- We’re working on some exciting enhancements to our methodologies to make results even easier to interpret and more meaningfully actionable for those with no formal Statistics background. Stay tuned!
Here’s the full response if you’re interested in reading more: http://community.optimizely.com/t5/Strategy-Culture/Let-s-ta...
Overall I think it’s great that we’re having this conversation in a public forum because it draws attention to the fact that statistics matter in interpreting test results accurately. All too often, I see people running A/B tests without thinking about how to ensure their results are statistically valid.
Dan
Optimizely is a website optimization platform. We enable businesses to show the right thing to the right person at the right time.
Our first product makes A/B Testing easy. In 4 years we’ve grown to become #1 in the category with 7,000+ paying customers including Starbucks, Disney, and Salesforce. We're profitable and our revenue is growing 400% year-over-year. Last May, we raised 58 million dollars to allow us to continue building great new products.
Join us in our mission to empower businesses to make better data-driven decisions.
About the Job:
We have scale: 7,000+ customers (including 100 of the top websites in the world); 120 billion server requests/month You’ll work on our amazing visual editor - a cutting edge tool that is the core of our product - and will enable it to do even more awesome things for our customers Work in small teams, in a continuous integration environment; we’re fast and nimble: 1-2 deployments every day
About Us:
You’ll work on our amazing visual editor - a cutting edge tool that is the core of our product - and will enable it to do even more awesome things for our customers. We are also building a mobile SDK for both iOS and Android and are working on some great new products.
Requirements:
A track record of being able to ship product, independently and/or as part of a small team. We will leave out the long list of languages because we would prefer a great engineer to someone who knows some Ruby.
Bonus Points:
Passionate about front end technologies, you have a curiosity that motivates you to experiment and keep on top of technical trends Practitioner of progressive enhancement, and responsive UI design Good at debugging using dev consoles, firebug, firecookie, or any of your favorite tools Fearless - you’ll tackle a massive JavaScript codebase, and are willing to prototype crazy, awesome ideas
Perks:
Free city-wide Gym membership at any Crunch Fitness. Free Clipper Card to pay for your commute from anywhere in the bay area. Catered in-office lunch and dinner on weekdays. Full medical insurance with very low co-pay and deductible. HMO, PPO, and HSA options available. Full dental coverage including orthodontics. Full vision coverage including contacts. Dependents 100% covered for medical, dental, and vision. Unlimited vacation policy. 401k benefit. Top-of-the-line MacBook Pro or MacBook Air and 30" monitor. Working with a great team and having a huge impact!
If you're interested, please shoot me an email at Ryan.Jordan@optimizely.com!
That said, there are other folks who do need a little extra help so we built them an academy.
FWIW, two weeks ago we launched our Optimizely Academy to help folks like you go from zero to hero: https://learn.optimizely.com/hc/en-us
What we are optimizing here for is end-user experience and minimizing the chance they have a higher-than-tolerable response time regardless of the variation they see.
Direct link: http://pages.optimizely.com/CDNBalancingWhitepaper_GeneralLP...
Here is a video of the office in Amsterdam: https://vimeo.com/63432751 Come by and visit!
A great source for this data if you are looking to dive into it is http://trends.builtwith.com/analytics/Optimizely
Regarding the 1.3 billion visitors: we define this as the sum of all the unique visitors to each experiment that have been run through Optimizely. We purposely avoided calling them 1.3 billion "unique" visitors because some people will be bucketed into multiple experiments and across multiple customers. We purposely do NOT include people who were not bucketed as part of the experiment/control groups. If we did, this number would likely be much higher since many of our customers use our traffic allocation feature that let's them only sample a small fraction of their traffic to include in the experiment: http://support.optimizely.com/customer/portal/articles/58196...
I wholeheartedly agree with your point that many of the players who started after 2008 have far superior implementations due to the advancements in browser technologies which T&T failed to capitalize. We certainly stand on the shoulders of giants. We would not exist as a company without jQuery, Akamai, Google App Engine, and Amazon Web Services. Not to mention the fact that browsers are much faster and can execute JavaScript in real-time to manipulate the DOM before the page is rendered in the browser and shown to the user.
As for revenue numbers: one of the advantages we have as a startup is that we are not required to publicly tell the world how big we are. If Omniture knew how big we were they might actually allocate resources to try to improve their products.
You are right that many of our 2,800+ customers pay us far more than our highest listed price of $399.
As for top customers in the categories you mentioned we are thrilled to have Disney, Starbucks, Salesforce, Crate&Barrel, GoDaddy, Footlocker, Electronic Arts, CareerBuilder, and more. It's refreshing to see companies that are traditionally "sold to" using a enterprise sales approach who are eager to embrace the shift to more nimble products that "just work." Old school sales tactics are falling on deaf ears as companies shift toward purchasing more like consumers. The vast majority of our customers come to us through word-of-mouth. We are riding the wave of the consumerization of enterprise.
If you look back you can see we've been growing at a pretty steady clip every since we launched: http://trends.builtwith.com/analytics/Optimizely
However you are right that there does seem to be an inflection point in May 2012. I can only speculate why this happened. Here are a few possible reasons:
1. We got really lucky with a bunch of great press at the end of April including a phenomenal piece in Wired: http://www.wired.com/business/2012/04/ff_abtesting/ and a piece on CNN: http://www.youtube.com/watch?v=TEpsfQHNUxU
2. We announced our funding: http://blog.optimizely.com/2012/05/30/optimizely-surpasses-g...
3. Probably in some part because of #1 we started ranking very high for organic search terms. For example we are now the first ranked result for "website a/b testing" on Google.
Hope that answers your question!
I'm glad to see you are focusing on improving your response time. Unfortunately hitting live servers serving dynamic content in 5 locations is never going to be able to beat 100,000 edge-cached nodes in 75 countries serving static content when it comes to response time and reliability.
According to the link you provided it looks like Pingdom reports VWO response times as:
VWO Slowest Average: 352ms
VWO Overall Average: 207ms
VWO Fastest Average: 45ms
According to Pingdom, Optimizely response times are:
Optimizely Slowest Average: 95ms
Optimizely Average: 65ms
Optimizely Fastest Average: 19ms
Here is a head-to-head comparison of Optimizely vs. VWO including response time and feature differentiation: http://i.imgur.com/BRTxC.png
To clarify some of the points you might have misunderstood:
- Here is a demo showing the Optimizely visual mobile editor: http://www.youtube.com/watch?v=HucXw_egPRs
- Documentation of Optimizely one-click integrations: http://support.optimizely.com/customer/portal/articles/58188...
- The Optimizely API is publicly available at: https://www.optimizely.com/docs/api
You are right we don't currently offer ancillary features like heatmaps and usability testing. We've been focused on offering the best possible experience for A/B and multivariate testing.
All plans come with a 30 day FREE trial and a 100% Satisfaction Money Back Guarantee.
Our SLA is part of the Mutual Services Agreement contract we have with customers.