After spending well over $100k in SEO i can tell you sadly the most effective tactic has been spam. Yep, tons of links on sites, setting up gateway pages that are focused on particular keywords, spamming social sites with links, setting up blogs to target keywords, etc.. I've hired/fired about 5 different companies in India and finally found one that is working magic for us.
Google's algorithm is not a mystery - most 'SEO' people will try to sell you some snake oil. I was paying 10k a month to a top name SEO consultant (very popular SEO book, well respected) and never felt so cheated in my life. Beyond the basics of SEO like semantic structure of your HTML and URL's - SEO is primarily a popularity contest.
I would just like to clarify that I have never had a customer paying 5 figures a month that was unhappy with what they got from me. ;)
Also agree with you that spammy links can work better than what one would think if they attended Google's sermons about quality links. Though typically from a risk profile it is better to mix and match so whatever your promoting doesn't stick out as an outlier & get whacked at some point.
Google's current ranking algorithm is a mystery. Sure we have some good clues to go on but outside of Google's own info (which on at least one occassion I've considered to be a deliberate attempt to mislead, though I might be wrong) or industrial espionage the only way to determine the nature of the algo is to experiment (or use other's experiments).
Yes they have a heavy reliance on site authority and the external link structure but this is not all there is.
Good content is arguably what Google wants because it is how they mine their product (people) to be delivered to their customers. Whether their algo is optimised to deliver good content is a separate issue - you clearly think not.
Haven't you been stung with any of the algo updates?
What % of spam sites are wrapped in AdSense? Greater than 5%? Greater than 50%?
For all Google's talk of content quality, they are clearly willing to make some concessions with who they have as business partners and the type of "content" they are willing to fund.
It wouldn't matter if it was 100%. Google apply pressure via their ranking algorithm to encourage quality content which in turn relies incoming links and to a lesser extent well crafted pages. Basically Google don't primarily themselves judge the quality - that would be futile with so many pages, they allow the net to do it, that's the whole point of PageRank to attain a metric for quality without actually measuring the quality per se.
On the subject of "spam sites" - spam is unsolicited content delivered to you, that's not possible unless you're talking about popup/popunders. You actively seek out the sites.
Now if the top rankings on Google are unwanted poor quality sites and Google are advertising spamily and not flagging malware, etc., then I think you'd have a point.
Bloglines is now a scraped "answers" website. If I took our old Threadwatch.org and did the exact same thing would Google let it rank based on existing link equity? If not, what would be the difference? Ask is a bigger partner so the editorial judgements are not made against them the same way they might a smaller player.
Google has frequently exercised editorial judgement against some folks, while letter other folks get away with doing the exact same thing in bulk.
Further, those who are creating original high-quality content have real business costs. Google paying scraper sites like Mahalo and Ask to borrow your content & wrap it in ads means that you are sometimes getting outranked for scraped duplications of your own content. That drives down publisher margins and pushes marginally profitable publishers into losing money.
That said, Google wants to get big into television ads. And that is going to mean having better respect for copyright. To some degree as we see the Google business model change we will see their approach of "paying anyone to steal anything & wrap it in Google ads" (to soften up copyright) change to a model where the put themselves as a gatekeeper on DRM content & push the "official" sources of the media (and try to make a cut of the profits).
Slowly but surely the search results will fill up with official hotel sites, official music sources, official video sources, official ebook sources, etc etc etc ... with Google putting a big foot on the gas.
As that shift happens the longtail spam model will lose out on its profitability because it will be forced to compete with higher quality content that is automatically mixed into the search results. (The whole point of universal search was to allow Google to short cut certain types of information right into the core search results...as they start making money from micro-payments and such look for that trend to accelerate).
If you had to choose _one_ vertical to be in, would _you_ chose one where every competitor in it knew all the dirty SEO tricks in the book?
Anytime you sell something which is a higher end service or an abstract type of service what drives sales is word of mouth recommendations rather than rankings. This is true for the higher end of the legal spectrum, the higher end of the health spectrum, and the higher end for any b2b services sort of business model.
Those who are buying based on rankings will see the Google AdWords ads for $149 SEO and such...and get sucked into buying trash. Those who are even cheaper will get scammed by their domain registrar or web host.
It is hard to make much money in the SEO game teaching others or providing tools to other. It is much easier to make more as a publisher, as Patrick stated.
One of the reasons I haven't invested more into some tools area is just how much stuff is becoming freely accessible. Google's keyword tool is amazing, Google analytics is amazing, some of the competitive research tools are awesome & cheap.
When you get down to it, its hard to automate the link parts without running into issues. Automating some of the link research areas is still an area for innovation, but most the stuff around keyword research, keyword organization & general content creation have already been heavily commoditized.
Another area where some folks work is baking a semi-legitimate cloaking service into their SEO product & then making it seem legitimate by only working with huge brands who agree to minimum contract amounts in the 5 or 6 figures per month. That model works well for the SEO because they typically get paid by the click, but it is pretty crappy for the client because they end up reliant on a tool that can be pulled out from under them at any time.
Brokering links is a huge business as well...but its sorta a gray area.
He's an excellent writer and marketer, but it's not like he's privy to some magical SEO techniques that would explode SEO traffic.
I would be surprised if he uses SEO tools much himself. From what I've seen, he's very good at identifying small niches, creating good content, etc. Basically good SEO fundamentals, not cutting edge trickery.
Patrick's SEO abilities are notable on HN because he has managed to run a business quantifiably tied to SEO and search advertising. He's hasn't just "done" SEO. Lots of HN people can claim that. He's successfully taken a product to market with it. That's impressive.
Referring to Patrick as an SEO undervalues him. SEO isn't really his schtick. Engineered product marketing is. Engineered marketing includes SEO, but also A/B testing, metrics, optimization, and a grab bag of other "lean startup" ideas.
(Also, in case it isn't obvious, Patrick is popular here because he writes thoughtfully, helpfully, and often.)
While I'm impressed by the ability of people like Rank and Greywolf to identify and exploit new-found SEO techniques for short term profit, with the knowledge that Matt Cutts' team is going to be hot on their trail as soon as anybody notices, I'm _significantly_ more interested in Patrick's longterm sustainable business strategy.
While I might be prepared to go out on a grey-hat SEO limb for an idea of my own with only a marginal chance of success, I'm _much_ more cautious about risking Google blackholeing any great ideas I've got, or any of my clients' sites...
I guess what I'm saying is that "excellent writing and marketing" _is_ a magical SEO technique. It's not a $50k/month affiliate scheme or a $2k/day Adwords trick, but I suspect Patrick's BCC is a properly valuable business that someone might sensibly buy off him one day. I'm not so sure I'd ever consider buying a quick moneymaking website off Rand or Michael...
From the HN guidelines:
Please don't bait other users by inviting them to downmod you.
I eagerly welcome discussion on this topic.
His comment wouldn't lose anything, but rather win in terms of readability, by removing those 10 words ("and I'm going to be downvoted into oblivion for this"). Those words are simply unnecessary and unobjective. They don't provide anything except making the comment sound more sensational than it actually is.
There are good reasons for this guideline, and I think those reasons apply to everyone, including patio11 and pg.
I know this because he is a member of our community and we chat often, I tell him stuff that is off the record & sometimes he evolves the model & pushes my model of thinking in new directions.
If I had his programming skills my goal would to be able to retire by the end of next year. :D
Probably the only thing that holds Patrick back in terms of aggressively using SEO to make a ton of money is that he isn't primarily driven by making a boatload of money. Some people are, but he isn't.
Whether Patrick has the inclination to work on tools like this, I don't know. I think it'd be a good idea for him to write a book, too, but he pooh-pooh'd that idea on TechZing because there's just not enough money in it.
1) Providing value in a scalable fashion in SEO is hard, because Google has an incentive to make it hard. If SEO were only as hard as AdWords, Google would lose billions of dollars. Google considers virtually any repeatable process that improves rankings to be black hat. I don't enjoy having to joust with a giant team of PhDs who have infinite budget.
2) The SEO tools space is hard. Small-scale website owners are often very skeptical that there is positive ROI in SEO. (There is positive ROI in SEO. Crikey, if you learn one thing from me this year, learn that.) You have to do huge amounts of teaching to raise people to the point where they can begin to benefit from it. That starts, literally, at "What is a search engine?", because most website owners do not know, especially in small business.
For marginally savvier website owners, like the average HNer, you can skip some of the teaching and proceed directly to "I don't want to pay you money for this." You've been on the same HN I have for the last year, right? We talk a good game about raising $50,000 each of a dozen angels and valuations in the tens of millions and whatnot, but what happens every single time someone suggests raising prices past, oh, $20 a month?
http://news.ycombinator.com/item?id=1510986 http://news.ycombinator.com/item?id=1780348 http://news.ycombinator.com/item?id=1357592
SEO makes me thousands a month. I'm not interested in selling it for less than many hundreds a month, at the low end.
3) Enterprise sales is not my idea of fun. Selling SEO to people who really would seriously benefit from it -- companies which do a lot of transactions online -- requires a high touch sales process. I'm not incapable of that -- I do it for consulting -- but it isn't my passion in life, and it wouldn't scale easily without me building a sales force. That requires a whole lot of skills (hiring, managing salesmen, showing up promptly at nine to an office somewhere, etc) that I don't have experience with, natural aptitude for, or any reason to suspect that I'd be really good at.
4) What would it do, really? (Your "real" question.)
Broadly speaking,
+ SEO analytics: Hard to demonstrate value. Most obvious feature sets are well represented by free competitors. Significant competition from SEO training providers who like to throw in a bag of tools subscriptions to make the $N00 a month fee look cheaper.
+ Tools which purport to "do" SEO for you: if it works Google will call it black hat, if it doesn't work it is snake oil. I have seen an awful lot of snake oil.
+ Demand Media In A Box: Probably the best SEO tool I can think of writing -- automate topic selection and direct outsourced production of content for it. Let's hypothetically pretend that I both was capable of and wanted to write this: what is my incentive to selling it to people who pay little money and need lots of handholding when instead I could point it at any problem domain I wanted and make a million dollars each time?
5) What is the opportunity cost?
I'm pretty good at what I do: I make and sell software, and I do occasional consulting for other people who mostly make and sell software. Consulting is fantastically lucrative, and would (and has) helped me get together enough money to launch any software/service I care to.
I picked out one I thought was a good bet, and am busy implementing it. Ask me next December, but I am cautiously optimistic, in a way I have never been optimistic about SEO tools.
THAT SAID:
There may be a profitable, addressable market for SEO tools. There certainly is for SEO-related training/services -- SEOBook and SEOMoz are both doing quite well. YC has funded at least one SEO-related company.
I wouldn't touch it with a ten-foot pole, though.