Launch HN: RankScience (YC W17) – Automated Split-Testing for SEO | Hacker News Reader

Launch HN: RankScience (YC W17) – Automated Split-Testing for SEO

rankscience.com

121 pointsryanb84 comments

RankScience automates split-testing for SEO to grow organic search traffic for businesses. 80% of clicks from Google go to organic results and yet most companies don't know how to improve their SEO, or can't effectively measure their efforts to do so. Because of the scale of SEO and the constant change of both Google's ranking algorithm and your competitors' SEO campaigns, the only way to succeed in the long-run is with software and continuous testing.

We've built a CDN that enables our software to provide tactical SEO execution and run A/B testing experiments for SEO across millions of pages. Experiments typically take 14-21 days for Google to index and react to changes, and we use Bayesian Structural Time Series and Negative Binomial Regression models to determine the statistical significance of our experiments.

Our software is 100% technical SEO, and doesn't do anything black-hat, spammy, or anything related to link-building. One of our goals is to bring transparency and shed light on what is largely considered a shady industry, but is so important to so many companies' revenue and growth. In fact, If SEO didn't have such a bad reputation, we think someone else would have built this a long time ago.

SEO as an industry earned itself a stigma for being spammy: between buying links, creating low-quality pages stuffed with keywords and text intended for Google rather than humans, and the used car salesmen attitude that many SEOs have, many people have been conditioned to dismiss SEO as an invalid or illegitimate growth channel.

We're software engineers-turned-SEO's, who have previously consulted for dozens of companies on SEO, from YC startups to Fortune 500 companies like Pfizer. We previously shared our case study with HN, where we increased search traffic to Coderwall with one A/B test: https://www.rankscience.com/coderwall-seo-split-test

Ask us anything! We'd love to answer any questions you have about SEO, A/B testing, and RankScience.

Can you explain a bit more about the technical details of what the platform can do?

Does it integrate with web master tools and google analytics?

Does it help identify page populations that are stable and similar enough to be compared?

How are the different test versions sent to you guys?

Can we use our own CDN on top of your CDN?

What about dynamic pages that change over time?

How big of a sample size is needed to get statistically significant results?

Howdy etler!

> Does it integrate with web master tools and google analytics?

Yep!

> Does it help identify page populations that are stable and similar enough to be compared?

Yep!

> How are the different test versions sent to you guys?

For the most part, we own the design of treatments. Some of our customers have their own experiments, but that's a small minority of our customers. Happy to talk more in depth about this if you're interested! dillon@rankscience.com

> Can we use our own CDN on top of your CDN?

Yes, we encourage this too!

> What about dynamic pages that change over time?

We address this on a case-by-case basis, but our software is designed to be as un-invasive as possible in use cases that we do not expect.

> How big of a sample size is needed to get statistically significant results?

We have to consider several variables here. Sample size is a function of Google's crawl rate on your site, the number of pages you have, and the amount of traffic you receive.

Thanks for the response! I know the sample size question is hard to answer because of the number of variables involved, but the reason I ask is that the CoderWall blog post example mentioned 20,000 pages, which is more than we have so I was curious about what a rough rule of thumb range might be.

How does your pricing work? Do you have a free tier?

We charge a monthly service fee based on web traffic. No free tier at the moment. Send me a note at ryan@rankscience.com if you'd like to learn more!

I did email you... no response.

It really would have been helpful if you'd just shared it here so others could see as well!

Just replied :)

I've just checked my email and spam folder and it appears you haven't. My email was from six days ago, subject line "pricing" and the body includes "I asked on HN"...

A few questions about your product (I run SEO and digital marketing at LendUp - YCW12). This is very, very cool.

* What types of optimizations does it do/do you test?

* I assume tests take a while to run, waiting for Google to re-index etc. Do you essentially monitor rank changes, assume that Google has re-indexed at that point, and use that as the data to optimize on? Or do you have some smart way to monitor for when Google re-indexes?

* I'm again assuming here - that a user will plug in a few variations of things to test, and let your software test them? Or are there automatic optimizations the software tries to make?

* How many tests can one do in a given time period, without confounding test & control variants? It seems like they would take a while to run?

* Is there a good way to control for non-technical/non-content-based changes (e.g. external, links)? For example we get hundreds of negative links pointed at us per week. Do we just hope/assume that's not the cause of rank changes?

Hey there!

We test everything on-page. This usually includes CTR stuff, like titles and meta descriptions and paragraph text that Google extracts for the meta descript, headers, images, calls to action, even conversion rates.

Correct, we continually iterate based on feedback like rankings and clicks, and we're working on smarter ways to monitor Google's crawler too!

Users are as hands-on or hands-off as they'd like. We usually own the experiments, but some of our customers design their own too.

The number of tests depends on the number of pages and your traffic. Our largest customers have full factorial experiments with thousands of concurrent split tests every day.

Guys, The idea looks great. Congrats on the launch.

2 SEO questions:

1) a CDN means "thousands of websites on one IP address". Google doesn't like it UNTIL it knows, the IP belongs to a well-known CDN like cloundfront/cloudflare etc. Please comment?

2) An A/B test might look like "cloaking" to googles. How exactly do you run it? I assume, not in parallel, but "variation A then variation B" - correct? If yes, does it mean I have to leave the website UNTOUCHED for 21 days so the test results are not distorted by my other activities? (adding new content, internal links etc)

Thanks! Great questions

1) We actually work with CloudFlare/CloudFront etc if companies are already using them. In that case, we sit in between CloudFlare and your origin webserver. But if you don't already have a CDN in front of your website, our SEO CDN goes in front of your origin webserver, and we use AWS so as far as Google is concerned it looks like AWS. We also use different IPs for every customer.

2) We don't do cloaking - we're essentially testing somewhat small tweaks across groups of similar pages, and seeing if they have any impact on clicks or rankings. It is preferred if the pages remain largely the same, but they don't have to be totally untouched.

Thanks for the answer.

PS. whoa, exciting (the 'in-between cloudflare and your server' bit)

Looks great, congrats on the launch!

Two points of feedback:

(1) The idea of the product is clearly conveyed, but I'm confused on exactly how it works. The landing page mentions that title tags, headlines, meta-tags, etc get tweaked - exactly how is this done? Do I have to manually enter a bunch of alternative text, or are you using a big fancy thesaurus to switch out some key terms?

(2) How do you evaluate performance of the product? Solely through click rates, or by search rankings? How often do google search results get updated? In short, how do I know the product is working?

Thanks sgslo!

(1) We use both humans and software to generate experiments. For customers, it's completely automated.

(2) We look at all primary search metrics (clicks, impressions, CTR, and rankings), with clicks being our main metric. Search results get updated at a pace determined by Google's crawl rate, which varies per site depending on multiple factors including domain authority. We use bayesian structural time series and negative binomial regression models to measure impact and statistical significance to power our data-driven SEO recommendations.

As somebody who wrote an article about A/B testing title tags in 2011 before it was cool [1], this is an awesome idea. I've talked about SEO with many companies and coming up with the proper title tags and meta descriptions alone is often worth so much traffic for such little effort (once you get past the upfront cost of running the tests).

However, I think a critical aspect of SEO is thinking about an entire site holistically. Not only because certain signals are site-wide, but because a key aspect of SEO is deciding what pages of your website are "good" for SEO and which ones aren't, and then focusing on making the "good" pages better and not worrying about the "bad" pages. Good and bad in quotes because it is often quite a bit of an art and not a science.

How does RankScience play into this? You've nailed the on-page stuff but is there any world in which RankScience is able to talk about a site holistically and recommend which types of pages and content seem to be working most effectively (and maybe even suggesting pages that could be removed/de-indexed)? Or do you leave that to SEO consultants and you just nail the hell out of the on-page stuff.

1: https://www.thumbtack.com/engineering/seo-tip-titles-matter-...

It's a true honor to speak with an SEO A/B testing OG! :)

> How does RankScience play into this?

That's a really great point. Right now, our software either focuses on only pages which are good for SEO, or it runs less frequent experiments on the "bad" pages. We've had a few success stories of turning "bad" pages into good ones and having them become big revenue generators for our customers, but that's not common as I'm sure you can imagine.

Though using A/B testing to improve user experience or conversion rates is fine, I thought using A/B testing to reverse engineer the ranking algorithm was against the guidelines. Has this been updated?

From https://support.google.com/webmasters/answer/7238431?hl=en

> Best practices for website testing with Google Search

> The amount of time required for a reliable test will vary depending on factors like your conversion rates, and how much traffic your website gets; a good testing tool should tell you when you’ve gathered enough data to draw a reliable conclusion. Once you’ve concluded the test, you should update your site with the desired content variation(s) and remove all elements of the test as soon as possible, such as alternate URLs or testing scripts and markup. If we discover a site running an experiment for an unnecessarily long time, we may interpret this as an attempt to deceive search engines and take action accordingly. This is especially true if you’re serving one content variant to a large percentage of your users.

Next to this, the advice is to use rel="canonical" to avoid duplicate issues with Googlebot crawling your variations. When using rel="canonical" this should not show you how a variation influences ranking.

> If you’re running an A/B test with multiple URLs, you can use the rel=“canonical” link attribute on all of your alternate URLs to indicate that the original URL is the preferred version. We recommend using rel=“canonical” rather than a noindex meta tag because it more closely matches your intent in this situation.

This is link is related to Conversation Rate Optimization testing (like Optimizely). We don't do A/B testing on single pages, or do cloaking or anything of the sort, but we run experiments across groups of URLs, and then we sum up the results and run our analysis.

Also, our goal is not to deceive Google in anyway - a lot of our tests are related to increasing CTR (which is testing humans) and on-page times. (again testing humans) Overall we're trying to make pages better according to Google guidelines -- which leads to a better experience for users.

Some more explanation of how it works here: https://www.rankscience.com/how-it-works

My post was in response to the homepage copy:

> RankScience sits next to your website, making thousands of experiments to tweak your HTML in order to improve your page rankings.

This seems to me an attempt at reverse engineering the ranking algorithm. Is my interpretation correct? And if so, is this allowed / in scope of the guidelines?

We're not reverse engineering Google's ranking algorithm, so no. : ) Good luck to anyone trying to do that! (I'd recommend against it) Yes, you're allowed to make changes to your pages.

Correct me if I am wrong, but you are trying to determine the effect that many different tweaks have on Google's ranking. You are not explicitly trying to reverse engineer their search algorithm, but, implicitly, you are trying to build a model of their model, and to then use your implicit model to drive up rankings.

Of course, the final determination is entirely up to Google, but this seems kind of a risky game to play.

Google's algorithm is to reward high quality websites which add value to their users. Our approach to SEO is to help our customers define "high quality" in Google's ever-changing definition and serve their quality to Google's users. This is exactly what Google wants.

Thanks! Though my (dated?) concern is probably not all that common, you could think of adding something like "Split-testing to improve SEO is a perfectly legitimate marketing technique" in your FAQ/technical copy.

Thanks for your feedback! That's a pretty good idea!

Just want to chime in here and share that I'm a very happy customer. Ryan, Dillon, and Chad are some super smart guys who deeply understand SEO. I use RankScience for 7 Cups and Edvance360 and I've seen a huge ROI. They always find time to meet with me and my other team members. The advice and feedback they provide is worth the cost of the service alone. Highly, highly, highly recommended!

Thanks Glen!! :)

yaya, thanks!

We've been using RankScience for months at Suiteness. They've given our indexed pages a consistent boost every week.

Thanks Kyle! :)

Co-founder/CTO here. Please share any SEO questions!

How does your product compare to Distilled's offering? https://www.distilledodn.com/

Ours is automated and continuous, and there's is a self-service product. We like those guys though!

That's an enormous difference for me, for what it's worth

We allow self service but most of our customers use our team (currently 16 SEO Analysts and Consultants) to work with their team, helping ideate, manage and run tests on an ongoing basis. See https://www.distilledodn.com/learn-more/expert-insights/

I think it's fairly similar - I remember showing Ryan a demo of our tool back when he was at 7cupsoftea before RankScience got going.

We just published an update on our first year of data: https://www.distilled.net/resources/distilled-odn-by-the-num...

(Distilled founder & CEO here)

Why have you taken this approach to monetizing this SEO optimization method, instead of other options, for example PPC arbitrage?

Why A/B testing and not a more complex statistical hypothesis testing method?

The biggest reason is that there's so much more volume in organic search (80% of all clicks) vs paid search (20%), and the ROI on SEO is so high. There's also so many companies in the PPC space -- it's quite saturated. SEO is appealing to us because we think it's overlooked due to stigma.

SEO and PPC are really two very different games overall, but I have a ton of respect for people who are really good at PPC.

Thanks, just to be clear, I mean you generate the clicks via SEO and selling them on a PPC, lead, etc. basis.

Oh, I misunderstood. Good question. A lot of people have suggested this. The main reason we've avoided it is complexity, but it's something we might revisit. I'd also be concerned about customers not having an easy way to understand our pricing.

>> "I'd also be concerned about customers not having an easy way to understand our pricing."

That would likely to be best expressed relative to major PPC players like Google.

For example, "clicks are 20% cheaper than Google for the same keywords."

Anyway, just nitpicking, awesome solution to a big problem. Good luck!

Thanks for the thoughtful questions! :)

I think this is a super compelling idea. : ) Thanks!

> Why A/B testing and not a more complex statistical hypothesis testing method?

We like simple! We only take on extra complexity if it's worthwhile.

Does it work with French websites ?

Although we work with almost entirely English sites, we're starting to expand to non-English too.

Cool I applied with a French website. Happy to help you guys test it.

Yep, we work with all languages. One caveat though is we only test against Google metrics, so if you live in a country where Google is not the predominant search engine, we're of no help at the moment. : )

- Is there a minimize traffic to the website required to get statistically significant results? Or does it matter since you're trying to move the site up in the Google ranking? - Does site traffic affect the length of time it takes to test?

Good question -- all of the things you mention are factors. It's harder to run statistically significant experiments with really small sites that have few pages. The factors involved are: # of total pages, search traffic, and Google crawl rate.

seems like a hard nut to crack, considering there are constant changes to google algos that could screw with test results mid-test, and many other moving parts. I think what you are doing is great, have built similar tools in the past and always ran into issues due to so many moving parts, but im not a statistician.

Also what do you consider success? what if one title/meta combo gets a better rank but fewer clicks(ie; the bots like it but the users dont)? which one would you keep? I know many people like to brag about how many rankings they have but as an SEO for over a decade, i think traffic count is still king. in that id rather rank 11th for an extremely high volume term, then rank 2nd for a low volume one.

another thought... a blackhat site gets banned halfway through one of your tests and you move up based on no actions you took. thus you would get a false positive, no?

Dillon,

I have a proposal for you - please respond via my email

Ric AT myUserName

Hey Ric, we'll be in touch!

Congrats on the launch.

How do you check the effectiness of a SEO change? You check for Google rankings or traffic?

We look at Clicks, Rankings and Impressions. We use Google Analytics, Webmaster Tools, and another third party data source.

More clicks is obviously the ultimate goal, but sometimes we also have significant experiments where rankings or impressions going up are the key metric

by curiosity, what's the thrid party data source?

That's part of our secret sauce. :)

I consult with startups and tech companies (recently: SurveyMonkey) on SEO, so this is super-interesting... especially the automated aspect, which is pretty novel in this space.

What are some of the specific types of automated tests that you run?

Here's a case study with a simple example of a test: https://www.rankscience.com/coderwall-seo-split-test

* CTR on titles/meta descriptions * Any HTML changes * Design changes * Time on site

Thanks for the reply. Would you mind being more specific about what specific aspects of the "automated" tests are "automated"?

E.g., in the CoderWall example, is it propagating of the title tag change to all the pages in the test group that's "automated"?

At least to my ears, "automated" suggests that there are tests that are selected and run completely without human intervention. (Which is hard to imagine in the SEO space.) Is that in any way accurate?

Correct! Our customers don't have to lift a finger, and our software continually iterates on their SEO.

It seems like you integrate with the likes of Google (Webmaster Tools) and Cloudflare in ways not done before, so my assumptions here are probably a bit outdated.

Let's say you want to test a new title. You collect stats for a week, change the title, collect stats for another week. The two datasets are then compared. Am I close?

Does it mean that you can't AB test in parallel? If so, it's not optimal for time sensitive stuff like breaking news.

We don't run experiments on a single page, but instead run them across groups of similar pages. The easiest way to think about this is running a test across a "product" template on an e-commerce site. A site could have thousands of product pages, but they're all on the same template. We would grab a set of a few thousand product pages, and split them into control and variant groups, execute the change on the variant group, and monitor how Google reacts to the groups in aggregate over a period of time.

Depending on how many pages you have, we can run many tests in parallel.

Gotcha

Thanks

Thanks for the AMA! What's the best way to deal with Google algorithm changes? Also, at what point does site performance actually impact my SEO?

The only way to deal with Google algorithm updates is through experimentation. A data-driven approach is the only approach. : ) Staying abreast of what both Google is saying publicly and what the SEO community is saying also helps (ie Google announced they're cracking down on intrusive pop-ups for mobile sites)

Site performance is always relative to your competitors. Optimal server response time is around 200ms, and that's what folks should strive for, but I've seen sites have really slow pages and still get lots of traffic.

You guys mention < 25ms as the performance hit of having you guys in front of the client's application. Is there a more concrete SLA you guys provide? Do you have a sense of what that hit might cost in page rank (there's some theories out there that time to first byte is one of the features that search engines reward)?

Hey 0bfus, we're working on a fancy performance metrics dashboard and status page to answer these questions! Also, we've done extensive research on pageload speed's effect on page rank, and we found that <25ms of incremental latency does not have any impact. We're working on a case study for the latter!

I am bit wary of the idea, despite the fair intentions, but Google in particular tends to view anything remotely close to 'improving' search ranking using 3rd party 'software', as a threat to their own Ego-rithms.

I wish I could use a tool like this and know that it won't affect my rankings negatively, but I feel that Google will eventually punish sites that do a lot of A/B testing.

How does this compare with the other offerings in the space?

We focus on being automated! Our software never stops working to grow your search traffic, even when you're asleep or if you don't want to think about SEO at all.

What's the minimum amount of traffic to benefit from this offering. At what point should a startup seriously consider working with you

No real minimum - it's all case by case.

Would you consider donating some services to Zidisha (YC W14)? If yes, please email me – zack@zackburt.com. Thanks.

When you are just starting out with a website, what is the 20% of effort that leads to 80% of the results in regard to SEO?

Hey there! I know we just chatted on facebook about this, but just in case anybody else is wondering: we're working on this post. Stay tuned!

That looks great! How do you automate google's rank checking? Do you use proxies/swarm of VPS's?

We use Google Analytics, Google Search Console, and third party services to supplement keyword rankings data -- we don't crawl Google SERPs. :) Plenty of other companies do, though. Moz has great keyword ranking data.

How does your pricing work?

It's based on how much traffic goes through our CDN. If you signup on our site or send me a note directly ryan@rankscience.com I can give you more details

Hey guys! Congratulations on the launch! Very exciting!

We've been seeing some great results with SEO testing, and have recently had our biggest test result (in terms of revenue impact). Looking forward to hearing more of what you guys are up to. :)

I think the fact that DistilledODN, RankScience, Etsy, and Pinterest have all published SEO split-test results recently demonstrates the importance of this type of data-driven approach to SEO!

Best of luck with everything!

Tom, Distilled (Disclaimer: I run the https://www.distilledodn.com/ team)

Thanks Tom. Would love to buy you guys a beer sometime. : )

Reply on news.ycombinator.com