Rap Genius is Back on Google | Hacker News Reader

Rap Genius is Back on Google

news.rapgenius.com

242 pointstomlemon188 comments

Honestly, this is just saddening. Google needs to have a consistent policy towards these blackhat SEO offenders and enforce it for everyone. RapGenius bought backlinks to artificially inflate their pagerank. They knew they were cheating but they did it anyway. They got caught. And they were hit with a manual action on Google. Sounds good so far. But that's where it stops. RapGenius, using the connections they have from their multi-million dollars in VC funding, got back on Google and was able to avoid paying the price for their cheating.

Compare this to regular websites. Lots of smaller sites will end up paying a "whitehat SEO" firm to work on a "link strategy". These firms will claim up down and sideways that what they do is legal, ethical and follows Google's rules. But, what they actually do is either (1) create networks of fake sites to provide backlinks on certain terms to artificially boost the site, (2) place spam comments using bots on legitimate sites to do the same [not that this will thankfully no longer work well due to the latest Google algorithm update], or (3) pay legit sites to place backlinks to artificially transfer pagerank the same way that RapGenius did. Now, these other sites, when they get caught, they get a manual action or a smackdown. The difference? They have to actually pay the penalty. Arguing that they didn't know usually doesn't work. The penalty is LONG. They can't call on their VC firm to make calls at Google to give them a get out of jail free card.

I'd like to call on Google to create a public policy on how they handle these manual actions with some clearly defined penalties (example: 3 month manual action of 6 PR drop, etc) and to consistently enforce them across the board. That way a mom and pop site that pays a 'whitehat SEO firm' and gets caught doesn't have a worse time than a site like RapGenius that purposely engages in blackhat SEO who can use their VC connections to get out of having to pay a penalty in under 2 weeks. It's also sad that Google gave this get out of jail free card to a site whose entire business model is based around other people's copyrighted works which RapGenius doesn't have a license for, doesn't pay for, and publishes illegally.

This.

Google is a business, and there is a very good chance that they financially benefit from Rap Genius being restored. Other companies that suffer don't have that financial link, and maybe the best course is for them to raise VC funds (and posture themselves so that Google stands to lose if they penalize)

I agree with you in theory, but in practice it sounds like you expect google to act as if they care about things like fairness. They are a company. The bottom line is what matters. If you think it's unfair, then try to hit the bottom line by using other services and convincing your friends and family to do the same

How much revenue do you think RapGenius brings Google? Even if it was $10M/qtr that is a drop in the bucket for Google.

RapGenius brings Google $0 in revenue. RapGenius brings in $0 in revenue for anyone--it's not monetized.

Another way to look at it is: If X is a really good website in its category, and Google consistently puts X above other websites in search results for that category, the user experience on Google is better with X than without X. This will lead to more users using Google more, and contribute to all the effects that make Google money.

So even if Rap Genius is not bringing revenue to Google, Google certainly can derive value from Rap Genius that contributes to revenue. However, since the effects are indirect, it's not easy (though not impossible) to put a monetary value on this relationship.

Another way to look at it is if Google detonates start ups from VCs they co-invest with, will their own internal venture arm get squeezed out of some deals?

And if implicit or explicit risk protection (and back channeled information about potential upcoming risks) comes as a fringe benefit of direct investment, doesn't that "free insurance" allow Google Ventures to guarantee themselves more favorable investing terms?

Google has a monopoly, and where are monopolies the normal rules do not apply. Google can impact internet as a whole very greatly. That's why they should care about fairness, simply remember how Microsoft was forced to provide browser choice window and were penalized for pushing IE as a default option.

And even if, say, in Microsoft scenario they might benefited from using they monopoly position more than were penalized (can't for sure claim, because I don't know real numbers), it doesn't mean that companies should't not care about fairness. At the end of the day that is what competition suppose to do - force companies to think about fairness towards their customers.

No some one elses site takes over Rap Genus's top spot - might actually increase Google's PPC revenue if the replacement site doesn't answer the question.

I very much doubt Rap Genius' delisting hurt Google.

RG is in a commodity industry. Lyrics are lyrics are lyrics no matter where you get them.

Yes, RG is better than most other providers. However, I suspect that the number of Google users that actually care about their differentiation (like annotations) is vanishingly small.

I keep hearing people bandying about this "VC connection" baseless innuendo. How is that anything but baseless, cynical speculation?

Remember, we're talking about someone (Matt Cutts) who penalized Google's own Chome website for improper links. If there was ever a conflict of interest, that would be it.

The idea that Matt was doing RG a special favor through a16z or other investors is an extraordinary claim, and we're going to need more evidence than just gut feeling.

http://www.mattcutts.com/blog/disclosure/

    Also, I have invested in Perfect Third (the company that 
    makes the WakeMate), Zencoder, Cardpool, Tasty Labs,
    Drchrono, Grubwithus, PoundPay, Apportable, Mailgun, and 
    Parse. I have also invested in CircuitHub, PlanGrid, 
    Pixelapse, ZenPayroll, Trigger.io, Zenefits, and True
    Link Financial.

    I have also invested in Lowercase Capital (Lowercase 
    Ventures Fund I), Y Combinator (Y Combinator Fund II), 
    Lowercase 140, and Lowercase Spur.

Pretty reasonable to assume RG enjoyed a direct line to Cutts all through the entire thing.

X is an investor, therefore X must be directly persuaded by Y investor?

That's not any evidence that RG specifically had a direct line to Cutts, and that this actually influenced the outcome. Again, we're talking about someone who penalized Chrome. I don't know of any greater conflict of interest than your employer.

In an argument about someone's motivations, I am inclined to give them the benefit of the doubt in the context of their past behavior.

I'm inclined to believe that savvy investors would leverage their own investors and network to solve problems when they can. Especially when X is an investor in Y.

The incredibly fast timeline is pretty clear evidence of communication considering the standard response time is never and the standard unpunishing time is never and the standard result is basically "change your domain".

You're forgetting something.

As a search engine user, I don't care whether the search engine firm is "fair", whether their rules apply equally to everyone or not. It's not a country, it's a search engine.

You can say what you want about Rap Genius, but they're just about the only lyrics site that doesn't suck. As a searcher, I want to get to sites that don't suck. In the case of lyrics, that's Rap Genius.

Google has an interest to show me the best results. How they determine those results, and how much of that process is manual, is up to them. It's nice that they decided that penalties and virtual spankings are a good thing in the long run, and it's lovely that you think that they're too inconsistent and unfair about it, but I'm a user. I don't care. I just want the best results.

I own many old Vinyl records and often they come with the lyrics for all the songs on them. That's enough for me. I don't need funny annotations.

So, while Rap Genius is a cool website, I wouldn't say that all the other lyrics websites suck.

Even if the intent is actually enhance user experience, I think by making these decisions manually instead of algorithmically that a pretty reasonable assumption could be made of human bias inadvertently coming into play.

That's great in the short term. In the long term, having consistent and harsh punishments (whether manual or algorithmic) for SEO cheaters is very much in the interest of Google's users, because it preserves the integrity of the search results.

I don't think that punishment is necessarily effective in the long run either. If you are good at dirty SEO, you will accept your punishment on one site/business and restart with a different site/business, and hopefully last longer until getting caught this time.

With the goal of minimizing overall dirty SEO, Google shouldn't get caught up in an over-emphasis on punishment. Especially when the website being punished is actually the website that should be chosen, and they only use SEO to compete with other SEO.

> If you are good at dirty SEO, you will accept your punishment on one site/business and restart with a different site/business

It's not a black/white issue, sure for a very spammy website, it would work less (because that website is still out of the picture). But for a grey website like rapgenius, it would surely suck a lot more, because they would lose all their brand image, etc, which is quite something to build.

Disagree. The sanctions that Google applies indicate that it is unable to effectively index a free web and is instead relying on editorial control and favouritism.

Except that google is really an ad network, and if it's seen as favoring well-connected customers over not-well connected customers, then it's going to lose favor amongst those not-well connected customers.

Google's value is derived from its users, and its customers will remain if the eyeballs are there to justify the investment. The best way to keep those eyeballs is to continue to serve the best results, and in this case Rap Genius is often the best result.

This is already happening to some degree, however it has generally mostly been the smaller & weaker players who have been soaked first. And in many cases the web & search ultimately helped build up a lot of these players before later knocking them down.

A lot of people are quite apathetic until the impact is direct. They recognize their own usage of the service rather than empathizing with a story from someone who was soaked. Here is a perfect example from a smart, tech savvy person who is big on self-interest / own vertical, but entirely apathetic toward other verticals - and this post is less than 4 hours old https://twitter.com/blam/status/419537614014672897

The tricky part in all of this is that one man's life work is another man's spam. Everything at scale has some mix of original editorial vs user generated content vs scraped & reformatted content vs policing of the mix. Ultimately the entity with a built in audience will be able to win more and more verticals over time due to preferential placement. A lot of people are unaware of how the knowledge graph will keep extending to swallow a larger and larger portion of the overall query stream. Anything of significant value that can be structured eventually will.

One of the more alarming articles I have ever read was one from 2007 where the New York Times explained about Query Deserves Freshness coming about after Google found that their own new finance service didn't rank as well as it should have. That was the beginning of the end for the indy web. http://www.nytimes.com/2007/06/03/business/yourmoney/03googl...

The 2009 recession was the icing on the cake that cut Google's revenue growth rate & got them fired up on moving beyond direct marketing to focus more on brand, brand, brand.

As a user I care whether Google finds me the most relevant results from the biggest haystack or whether it editorialises those results - it's a big distinction. By that measure I'd say they definitely _shouldn't_ be penalising sites for finding loopholes in their algorithms but should simply close those holes up.

Whether a search engine presents "the most relevant results" is a subjective determination that can only be made with certainty on a per-user basis.

I'm confident in this assertion because people have, at various times, claimed that other search engines provide them more relevant results, yet for me, those search engines almost always failed to even match Google, and never exceeded it.

This was also the case pre-Google -- e.g. lots of people swore by AltaVista, but I almost never used it, because it never seemed to work well for me. I actually remained primarily a WebCrawler user for quite a while, right up until settling into Google in the 1999~2000 timeframe.

I don't want to be limited to search engines that provide what some bureaucrat has decided are the most relevant and/or neutral results. I want to be able to choose the search engine that provides the most relevant results for me, and I have seen no evidence that what's relevant for one person is necessarily relevant for everyone else.

> what some bureaucrat has decided

That is precisely my point.

(And if you are looking for personalisation, an algorithm isn't necessarily counter to that but a global smack-ban on behaviours most certainly is.)

Government bureaucrat. I don't care if Google, Microsoft, or whoever else has humans who directly modify search results. I'm free to choose among the competing search engines based on the value they provide me.

Any external notion of correctness imposed industry-wide will destroy that choice.

I wasn't talking about anyone outside google, just critiquing the game being rigged internally, but it's maybe interesting you brought that in.

On the one hand that could be criticised as an illusion of choice in the market (behind the curtain there is often ownership across entire industry sectors, regardless of 'competitors' within a sector). On the other, governments quite correctly get a say in any case (what do we representatively govern otherwise - the alternative is to cede governance to global corporations). Google is particularly sensitive to the latter as it has a huge de facto monopoly and has become part of our infrastructure - it is absolutely a huge target for regulation wherever it has traction.

Talking of competition, is there anything viable in the shape of an open source effort where the algorithms and indexes could be crowd-managed - perhaps indexing via browser plugins?

> has become part of our infrastructure

I think a comment[1] I wrote a couple months ago about Facebook is relevant here.

Infrastructure is what services are built on. It's a prerequisite, not the end result, and its absence is extremely costly.

Google is not infrastructure[2]. Like Facebook, it could disappear tomorrow, and we'd all just switch to other search engines. I assume Bing would immediately pick up most of the users, either directly or via Yahoo. I wouldn't like it as much, but I would not be materially harmed.

Google has done nothing to make it harder for someone else to build a search engine except raise the bar for quality. It has a better product for most people, so it has the most users. That's how it's supposed to work.

Using Google's popularity as an excuse for regulation is nothing but a demand for mediocrity. It's saying good things aren't allowed to exist, and that people shouldn't have the right to choose (or make) better products.

It's saying "CONFORM!".

[1] https://news.ycombinator.com/item?id=6621525

[2] Well, Google does offer something closer to actual infrastructure of course, in the form of their App Engine/Compute Engine offerings, but they're even less dominant in that field.

I think what google provide has become societal infrastructure, particularly wrt their monopoly position. That someone else would take their place (however badly executed) is evidence of that. We use it for studying, planning journeys, shopping, finding news, converting measures etc etc. In the process it's taken a position of political and economic importance. It makes a difference to us if google suppresses or boosts certain information and can influence our electorate during an election (e.g. many countries ban publication of exit polls); it affects the courts if certain information is made easily available (e.g. can prejudice trials); economic data and editorial affects the financial markets - and so on. If one player dominates this sphere, it is problematic - indeed it's why there are limits on media ownership. That's nothing to do with a demand for mediocrity but a demand for diversity, which is already well established. And yes companies very much need to conform to the countries they want to do business in - that's a good thing for the citizens of those countries.

This is what I just heard you say: Regulation is needed because people might exercise free expression.

Do you have any idea how evil that sounds? If that is the argument, this is my answer: I will take up arms and die fighting to prevent the realization of such a goal.

Media ownership limits are justified only because of a resource scarcity. They do not apply to Internet publications, and with search engines, all I have to do is type in a different domain name. There's no spectrum to be monopolized.

That's disingenuous - I was talking about the normal limits on free expression (shouting 'fire' in the theater) and gave some concrete examples. Surely you're not suggesting it is viable for a foreign company to undermine a country's judiciary in the name of it's 'free expression'? (And let's not pretend that Google is following a social or cultural mission rather than a business one.)

Media ownership limits are about market monopolies, not scarcity, newspapers being a prime example.

Those are not "normal limits" in the United States, and I do not believe they should be "normal limits" anywhere.

If it undermines a country's judiciary for information about crimes to be published, that country's judiciary needs to be undermined. Once information has leaked beyond law enforcement officials, there is not and should not be anything anyone can do to prevent its spread.

I don't care what mission Google is on. Rights are not dependent on motive.

Newspaper ownership limits exist only in relation to broadcast station ownership. There are no separate limits on newspaper ownership.

What you've said is correct, but it's also a bit myopic.

The whole point of Google's search algorithms is to "curate" results using algorithms and metrics based on the behavior of users on the Web. In a sense, the curation of content is crowdsourced, drawn from the activity of the entire network.

When Google takes actions that inject nepotism into this system, it sets a dangerous precedent. I don't want individual people at Google arbitrarily prioritizing what content I should be seeing. (There are other sites for that kind of curation.) I want the collective actions of Internet users to decide this via Google algorithms. There's an important difference.

I search for "[song name] lyrics" and I get a site that has the lyrics. I've never thought it sucked. I don't care how many banners on the side, just that the words are there. And I prefer an "expanded" view with all the [Hook] type stuff replaced with the actual words, rather than the "more correct" RG presentation.

Heck, sometimes I have to click more to get what I'm looking for with RG. After trying a few things, I was able to reproduce this just now with a search for "forgot about dre lyrics" (no quotes). This is the second result I got: something google shows as "rapgenius.com/...dre-forgot-about-dre/Eazy-es-ice-cubes-and-docs-the-s..." that does a redirect to http://rapgenius.com/Dr-dre-forgot-about-dre-lyrics#note-252... which pops up an annotation over the lyrics. That's not what I was looking for.

All that to say: it's not clear to me that RG really sucks less for "[song name] lyrics" queries.

> All that to say: it's not clear to me that RG really sucks less for "[song name] lyrics" queries.

1000x this.

I think Rap Genius is great for "song meaning" queries, but their interface is just as distracting (if not more) than other lyric sites that "suck."

Actually I'd be surprised if Rap Genius is any good for "song meaning" queries. Will it give you information? Probably. Will that information be accurate? I'm skeptical.

I've done a few spot checks on Poetry Genius, at least, and the annotations have consistently been either wrong or irrelevant, even for works that are well documented (e.g. The Waste Land).

My guess would be that if Rap Genius users aren't getting it right when large bodies of secondary literature are available, they can't be doing very well with pop music lyrics either. I could certainly be wrong, though.

They have verified artists program (http://rapgenius.com/verified-artists) where artists pop into explain the lyrics themselves. It might not be perfect, but there are only so many players in the "song meaning" arena & I bet RG is pretty high up there.

When and how do you search for lyrics and what makes RG better than the rest?

> As a search engine user, I don't care whether the search engine firm is "fair", whether their rules apply equally to everyone or not. It's not a country, it's a search engine.

Google wants its results to be the most relevant, not contain the best cheaters. Allowing cheating would set a bad precedent eventually leading to deterioration of search quality.

Read the article.

They are back because they did a lot of work finding all these bad links and using Google’s disavowal tool to kill them, not because of their VC funding or whatever other influence you think they have on Google.

Nice try.

Removing 200 links out of nearly 200,000 links in a holiday week is not a lot of work. In fact, Google has told others waiting to be reconsidered that the wait time is multiple weeks. https://twitter.com/Marie_Haynes/statuses/418734227228606464

Further, some sites which submitted reconsideration requests before RapGenius was penalized have not received a response. And in many such cases those other sites:

had far fewer links, removed a far greater quantity of links, disavowed a far greater quantity of links & were involved in something that was nowhere near as overt/intentional on the behalf of the webmaster (I've seen cases where the notices came after a payday loan hacker mixed in someone else's sites with their own links to create cover for their work)

Not to argue with Aaron Wall on an SEO topic, but you read the same article I did. There were 200,000 total links to Rap Genius and most of them were legit. Only a small subset analyzed ended up being suspicious.

They got 200 webmasters to remove their links, a few didn't, and the rest ended up being "site scrapers" who steal other people's sites. They just added those to disavow. Seems like they acted very quickly to rectify every suspicious link they could find.

But yes, they got special treatment in having the manual penalty removed after only a few days. But it's a legitimately popular site. You'd expect Google to do that since its harming Google's own users during the penalty.

A very well known and funded business gets preferential treatment. BOO-HOO. Do you think Bill Gates waits on a line at the bank? At the mayor's office? Anywhere at all? In real life if you are powerful, you get respect.

There is no such thing as fairness. It's a paradigm of the feeble minded. Most likely a defense mechanism - "look guys, he cheated on the exam, let's tell the teacher!".

Someone took the Godfather to heart.

So does this mean when Google talks about these same topics & uses the same sort of black-and-white framing (while encouraging "feeble minded" spam reporting and so on) we should consider them to be intentionally lying?

If no, please explain.

If yes, an important follow up on that front: how dystopian is it that the company in charge of organizing information requires spreading misinformation to succeed?

But if they post clearly defined limits and penalties, then that brings them into the territory of offering a "terms of service" for which they can potentially be sued.

So on many levels, keeping it vague and saying simply, in so many words "just be good, don't do anything abusive" like many services do for high-volume APIs (e.g. YouTube, last I checked) -- makes for a pretty good, common-sense fit in this case.

It also puts the emphasis were it should be (in my view): not on "don't do X or Y, OR ELSE" but on "look, just be ethical, and use your head, please."

It's just too bad for RG that they didn't head Google gentle and clearly (enough) worded advice, first time around.

Banning Rap Genius also made Google look bad. I bet 100,000s of people search Google for "rap genius get lucky" and similar searches every day, and Google was not displaying the results they clearly wanted. In this case, the Google search result page was clearly "worse" for the ban. Google harmed itself also in this ban.

Which does not happen when they ban your herb garden equipment website.

It looks as brand related searches were not that significant until the Google penalty happened.

http://www.google.com/trends/explore#q=rapgenius%2C%20rap%20...

According to Compete.com about 5% of their searches were brand-related queries https://app.compete.com/website/rapgenius.com/

SimilarWeb also shows most their top queries were unbranded http://www.similarweb.com/website/rapgenius.com

Alexa puts the branded-traffic number closer to 2% http://www.alexa.com/siteinfo/rapgenius.com

(All these tools have errors, but sites like Amazon.com or eBay are more like 15% to 25%+ on the branded traffic front http://www.alexa.com/siteinfo/amazon.com http://www.alexa.com/siteinfo/ebay.com ).

Do you have any evidence "VC connections" had anything to do with anything? Or are you just making shit up?

OK then, I'll just put you down for "making shit up".

It is digital and everything can be traced back - almost. But it is still the world of rich and famous. A similar & more severe penalty was lifted off the most popular Indian ecommerce site - flipkart. I remember meeting their head of marketing in 2011 who had no idea of the penalty which hit the site probably 2009-10.

As a user, yes I would like this site to rank up just so RG fans are saying they like RG result in google. But it is so unfair Google sends cryptic messages on webmaster for reconsideration requests but can entertain a direct call and lift off penalty. The least Google could do is to be transparent (fairness is hard)

This is great for Rap Genius.

But this truly saddens me as well. I cannot for the life of me understand how they were able to fix ten-of-thousands of backlinks and other shady practices and get back in the good graces of Google so darn quickly.

Frankly I do not know a single other legitimate(+with good traffic) site who was able to get back in the good graces of Google after algorithm updates without spending many many months researching any possible reasons to be penalized and fixing it themselves after repeatedly asking google for help to no avail.

Could it be that Rap Genius was special because it has Andreessen Horowitz?

Sure seems that way.

I am not going to stop using Google, but I am inclined to use them lesser now.

I would like to hear why, how rap genius was fixed this and what this means for other people who get penalized from Matt Cutts though!!

They're back on Google, but keep in mind that they're seemingly not in a great position. Googling "justin beiber lyrics" doesn't show Rapgenius. Neither does "lady gaga lyrics" or "tupac lyrics". In all three cases, azlyrics.com has the top spot.

Edit: the reply is to the now reacted comment:

   > They're a startup, and this was a problem which was either going to kill them or be surmounted quickly. Naturally, they surmounted it quickly.

   Google laid out a clear plan of action for them. Is it hard to believe they finished it quickly?

Yeah, most of the other sites I refer to were also start-ups by almost any definition.

They did not quite finish their action plan. They got away on reasonable doubt.

This is the first time I am seeing this defense considered by Google.

I willing to bet any other non-vc funded bootstrapped start-up would never be afforded this consideration by Google.

This rant is not a judgement on Rap Genius, I am actually glad they maneuvered around this.

I am judging Google.

They have killed other start-ups for reasons such as other sites farming and duplication their content more vehemently. Reasonable doubt has never been considered for anybody else and most likely will never be hence forth.

I just want to add that this attitude doesn't even reflect poorly solely on Google Search. Adsense is another product that if you get kicked out of, you have zero recourse.

What about that gmail fiasco a few months ago? Users got blocked out of G services for not providing their real names and had no recourse to get their data for several weeks/months.

Just goes to show that in several cases it's not what you know but who you know.

I have never seen Rap Genius on Google, even before all of this. I've seen Metro Lyrics, AZ Lyrics, Song Meaning, etc. I've gone through pages of Google results because every lyrics site sucks, still haven't seen them. Everyone says Rap Genius is excellent, but I've never heard of them outside of HN. I find that odd.

You have the causality wrong. Rap Genius has Andreessen Horowitz because Rap Genius is special.

In this case, the user experience for a Google user is worse when Rap Genius is removed. Google has some incentive to be more active in fixing the problem there, especially since Rap Genius was cooperative, technically competent, and aside from their "rapper swag persona", genuinely nice guys.

Can someone clarify for me, again, what is so special about Rap Genius? It's just a bunch of stupid rap lyrics. So what? In fact, how are they not being sued for copyright by the owners of all the lyrics they're posting, anyway?

I run across Rap Genius now and then and have always considered it to be spam, just like yahoo answers, ehow, and mahalo pages.

http://news.rapgenius.com/Marc-andreessen-why-andreessen-hor...

    Rap Genius [...] and have always considered it to be spam

That's pretty absurd...

Can someone clarify for me, again, what is so special about Rap Genius?

Why? You've no doubt read 10x reasons people here have said so (in the other threads). Do you just want to argue? If you don't agree, fine, but clearly a large and vocal group does. So what? Move along - find other things to care about.

In fact, how are they not being sued for copyright by the owners of all the lyrics they're posting, anyway?

They have had some legal troubles, but are trying to sign licensing deals http://www.nytimes.com/2013/11/15/business/media/rap-genius-...

However, if their licensing deals are anything like what other sites have to pay for lyrics, it's going to significantly (entirely?) cut into their profit margin.

Sorry to say, but your assessment of the site is 100% incorrect. It STARTED as a Rap Lyrics website, but it quickly expanded to be an ANY lyrics website (and they will probably expand beyond that)

The main driving force behind it is that anyone can edit the lyrics, and even better, anyone can ANNOTATE the lyrics to explain what certain sections mean.

Let's take for example the lyrics for Mumford & Sons' "The Cave": http://rock.rapgenius.com/Mumford-and-sons-the-cave-lyrics

I looked up this song one day trying to understand more of the allusions made in the song; I know the title itself alludes to Plato's 'The Cave' and wanted to delve deeper. Because of this site, I can see that Verse 3:

> So come out of your cave walking on your hands

> And see the world hanging upside down

> You can understand dependence

> When you know the maker's hand

is an allusion to St. Francis.

Most of the time when Rap Genius ISN'T the top result for lyrics when I google, I facepalm and go directly to their site.

I should emphasize here that this overly positive post isn't intended to condone their shady, spammy blackhat SEO tactics, but damnit if they aren't the BEST lyrics site by a fair margin.

> even better, anyone can ANNOTATE the lyrics to explain what certain sections mean.

Maybe I looked at the wrong lyrics, but I saw precious few explanations, instead most lines had jokes and/or totally unrelated pictures attached to them. Often enough the jokes is to simply restate what was said in rapper slang in plain english, which is funny for exactly the first time you read it, then it's just spam.

All I know is that they somehow come up in my google results fairly often and usually unrelated to the context of my intended search and it always felt like it was SEO'd junk crapping up my results. I added them to my google filter to keep them from showing up in my searches anymore, along with all the inane commentary from their community discussion underneath each song.

If you think Rap Genius is stupid try using any other lyrics site. That's what is so special about it.

Is anyone else getting more and more suspicious of the scorn that people have been gratuitously heaping on RG? Like, did they run over everyone's dog?

I actually avoid using them because the design is hard on the eyes and I find the way they present annotations to be annoying. I don't think there's anything special about them at all, especially when you consider that the idea is completely unoriginal.

But I'm not trying to use any lyrics site. Rap Genius garbage just always popped up on my (to me)-unrelated searches. People can respond to me with how they're not spam or they're not as spammy as other lyrics sites, but the point is that they pop up in the same frequency and same context as every other link-farm/content-mining crap-fest on the internet, like ehow, cha-cha, yahoo answers, mahalo, paidcontent, and so on.

> Google has some incentive to be more active in fixing the problem there

That's exactly how I feel too. Though the incentive for google is very unclear, the lack of clarity is explored further below.

> You have the causality wrong. Rap Genius has Andreessen Horowitz because Rap Genius is special.

Not saying Rap Genius is not a special lyrics start-up. Not even saying that it would not have been a good candidate for any VC financing them.

Though, Definitely saying that the special case leniency shown by Google sure make it seem like having Andreessen Horowitz is the only thing different between Rap Genius and scores of sites before with this and similar situations.

> In this case, the user experience for a Google user is worse when Rap Genius is removed.

And its not when other good sources are removed because they were link bombed or their content was scraped more than their competitors?

> Rap Genius was cooperative

Find me one person in their situation who would not be. It was to either co-operate or die.

> Rap Genius was technically competent

How?

> Aside from their "rapper swag persona", genuinely nice guys.

I am judging Google.

I have nothing against Rap Genius founders, nor their "rapper swag persona" and I'll believe you when you say the are nice guys.

Having worked for large ftse100 publishers who had sites penalized in algo updates its stinks that such an avowedly black hat site gets of with a slap on the wrist.

On a similar topic, how do affiliate linking programmes even work anymore? Surely Google would classify all affiliate programmes as paid links. Even big ones such as Amazon Affiliates, Commission Junction, Affiliate Window are all being plugged by affiliates, and the vast majority of those affiliates are using templated sites or stuffing keywords in blog posts that are largely irrelevant to the product or service they are plugging for affiliate fees?

Aren't all affiliate programmes technically bad for the publisher after Penguin?

Usually they are paid be purchase, irrespective of the number of clicks.

This wasn't an algorithm update.

If the folks at Google can figure out how to detect this sort of gaming more reliably, that may end up in a future algorithm update, and no amount of pleading will matter for anyone affected by it - they'll have to fix the problem or continue to suffer the consequences.

The algo is meant to detect such manipulative links and devalue them.

> Frankly I do not know a single other legitimate(+with good traffic) site who was able to get back in the good graces of Google after algorithm updates without spending many many months researching any possible reasons to be penalized and fixing it themselves after repeatedly asking google for help to no avail.

Well there wasn't an algorithm update, there was a manual action. Google reversed the manual action after RapGenius stopped breaking the rules and cleaned up its mess. If this happened to you there would be a note about it in Webmaster Tools and you would be able to ask Google to take another look. I'm sure RapGenius had all hands on deck making sure they were up to snuff.

Complete and utter bullshit.

That they blatantly employed blackhat SEO techniques and are back on Google within a matter of days is sickening.

Think of all the sites that are penalized in various ways for similarly shady SEO practices and have to spend _months_ to regain their previous position.

Hopefully Bing et al can capture some marketshare and bring some degree of neutrality to the search engine landscape (i.e. not have every website owner on the planet beholden to the beast).

I do not understand why people are offended that they came back quickly AFTER doing a bit of penance.

Should their investors not have made calls? However, it should be noted that they did their part.

One of my favorite quotes is this: "To save a drowning man, he must first give you his hand"

If the Rap Genius guys were incompetent and did not do their part in removing the links so quickly (see main story for technical details) there would be little or nothing their investors would have been able to do about that.

BTW, it also helped that people really found Rap Genius useful. I'm sure many searched had rapgenius appended to it. So the lesson is, make your app so good that when Google delists you, it will make them look bad.

Thankfully?, there is also a precedence for other offenders to use. if you can detail you have atoned for your sins and Google insists your you must do a certain time, you know where to turn to.

BTW, this is Hackernews, and I look forward to the discussion of the (de)merits of how they scrapped and analyzed 177,000 links and not espousing of anger that they survived the punishment.

I'm sure Mahbod (my favorite) and co have learned that you do not go 'daaawging' when you are in a hole. I wish them success.

Just to hijack on one of your points: "it also helped that people really found Rap Genius useful."

This is painfully true, and I know it has been discussed in prior discussions. Rap Genius is without a doubt THE BEST lyrics site available, hands down. Even if the other sites weren't ad-ridden scammy buggy shitty websites, if they got their acts together and actually TRIED, Rap Genius would still be top dog.

I don't care for the attitudes or personalities of the people in charge, and I am absolutely disgusted that they felt the need to scam their Google rank, but I am willing to see that some infractions can be rectified through penance, and am happy to know that they will not be forced to shut down because of this fiasco.

Because it is a double standard. There are thousands of sites which get blacklisted by Google. I'm sure all of them fix their errors really quickly once they realize they are on the black list. And none of them get back on Google nearly as quickly as rap genius did.

If Google applied these rules consistently with all websites, I would have no problem with it, but they don't. This breaks the myth that Silicon Valley is a meritocracy - as it shows that the connections you have are as important as raw merit, if not more

In life there is always a bias. When that bias is against you, you should learn how to fight back.

This is the PERFECT time for anyone who has done penance and is still being blacklisted by Google to document it all and go to the press.

There are A LOT of people/journalists waiting to jump on a story that shows Google has double standards.

But first they must be willing to do their part by documenting these facts and reaching out the journos. It will not come to them.

Like I quoted "to save a drowning man.....

The only problem with this is that there's no real course of communication with google. They make it a matter of policy to not provide customer service.

"in 2003 when Denise Griffin, the person in charge of Google’s small customer-support team, asked Page for a larger staff. Instead, he told her that the whole idea of customer support was ridiculous. Rather than assuming the unscalable task of answering users one by one, Page said, Google should enable users to answer one another’s questions." http://www.wired.com/magazine/2011/03/mf_larrypage/3/

Have a like, sir!

Fairness does not exist in real life. Screaming "unfair" will not help you achieve anything. It's such a ridiculous situation for me. All these guys are sitting on the line, raging at real life because popular culture brain washed them.

I can't believe I'm saying this, but

What a bunch of stupid nerds!

What about the other thousands of people that are likely waiting on their request?

And many of those little sites have the excuse of ignorance - they read a dodgy SEO page or hired bad SEO people.

RapGenious had no excuse.

Still, I'm glad they're out of penalty now.

Exactly. Everything about how Google handled this is wrong, in my opinion. I couldn't care less about RapGenius - they're just a symptom of the underlying problem which is we don't have any real competition in the search engine market, and we're paying for it with unfair antics like this. Longer thoughts here: http://peebs.org/2014/01/04/we-need-viable-search-engine-com...

Very much so. Google's decisions are starting to have impacts on other companies' bottom lines which easily run into the millions of dollars.

Just as a thought experiment ... how tempting do you think it would be for Google to do the following:

1) Run a query to identify all companies that cut their AdWords spending over 50% over the last year.

2) Look at which of these companies are now getting significant numbers of click from organic results.

3) Find out what is causing them to rank high organically.

4) Penalize them using a generic message and refuse on principle to answer any inquiries as to why.

Apart from (anec-)data that seems to be pointing towards this already happening; should we be OK as a market with this approach? Especially when considering that the rules are so vague that almost every site is guaranteed to break some of Google's guidelines?

>Apart from evidence that seems to be pointing towards this already happening;

Please provide this evidence. I am getting a bit tired of this misinformation. Organic search and paid search are silo'd. To say that Google favors advertisers in the organic results or the other way round, is simply not true. It would destroy Google's credibility.

>how tempting do you think it would be for Google

It approaches conspiracy thinking. I love that, but I think HN is not the place. You are basically accusing a company of a very evil act, without proper evidence. Do you realize that many Googlers frequent this site? It would be a shame if all they get to read are conspiracy theories and baseless accusations.

>Especially when considering that the rules are so vague that almost every site is guaranteed to break some

Hogwash. Their rules are very clear and succinct.

https://support.google.com/webmasters/answer/35769?hl=en

Maybe if you want to spam or manipulate, then these rules are vague. Maybe when you don't read the guidelines then they are vague. I bet that if you can point to a rule that almost every site breaks, without them knowing, then Google will adjust that rule to be more clear.

You can not please them all.

>Penalize them using a generic message and refuse on principle to answer any inquiries as to why.

You can certainly not please the spammers that were caught with their hands in the cookie jar and then take to the internet to say that Google is unfair, and that the big brands get away with anything.

In other words- "They just wouldn't!"

> Their rules are very clear and succinct.

If I think of a lot of things my users would search for and try to create valuable information for them concerning those queries... is that valuable content or are those doorway pages?

If I give when someone few months of my service for free when they write a nice article with a link to my site... is that good customer service or a link scheme?

If I create a press release on PR Web and link to my service with an appropriate keyword ... is that participating in a link scheme?

None of these cases are all that clear cut and I for one am not comfortable with Google being the judge, jury and executioner.

> It approaches conspiracy thinking.

That's not the point. The point is that Google is a massive company wielding massive power.

When it comes to fighting a penalization their position is that they don't need to defend their decisions, implying that they can be trusted not to abuse their position and hence deserve to have the final say, legally, over these decisions.

I don't buy that and think it's about time that contesting Google penalties in court becomes a regular thing.

> If I think of a a lot of things my users would search for and try to create valuable information for them concerning those queries... is that great content or are those doorway pages?

  Create a useful, information-rich site, and write pages 
  that clearly and accurately describe your content.
  
  Think about the words users would type to find your pages,
  and make sure that your site actually includes those words
  within it.

  Doorway pages are typically large sets of poor-quality 
  pages where each page is optimized for a specific keyword
  or phrase.

  Some examples of doorways include:

  Having multiple domain names targeted at specific regions
  or cities that funnel users to one page
 
  Templated pages made solely for affiliate linking
  
  Multiple pages on your site with similar content designed
  to rank for specific queries like city or state names

>If when someone writes a nice article with a link to my site I give them a a few months of my service for free... is that good customer service or a link scheme?

  Avoid tricks intended to improve search engine rankings. 
  A good rule of thumb is whether you'd feel comfortable
  explaining what you've done to a website that competes
  with you, or to a Google employee. Another useful test is
  to ask, "Does this help my users? Would I do this if
  search engines didn't exist?"

  Any links intended to manipulate PageRank or a site's 
  ranking in Google search results may be considered part of
  a link scheme and a violation of Google’s Webmaster
  Guidelines. This includes any behavior that manipulates
  links to your site or outgoing links from your site.

  Large-scale article marketing or guest posting campaigns 
  with keyword-rich anchor text links can negatively impact 
  a site's ranking in search results.

  The best way to get other sites to create high-quality, 
  relevant links to yours is to create unique, relevant
  content that can naturally gain popularity in the Internet
  community. 

  Effectively promoting your new content will lead to 
  faster discovery by those who are interested in the same 
  subject. Avoid: attempting to promote each new, small 
  piece of content you create; go for big, interesting 
  items. Avoid: involving your site in schemes where your 
  content is artificially promoted to the top of these 
  services. Avoid: spamming link requests out to all sites
  related to your topic area. Avoid: purchasing links from 
  another site with the aim of getting PageRank instead of
  traffic.

> If I create a press release on PR Web and link to my service with an appropriate keyword ... is that participating in a link scheme?

  Links with optimized anchor text in articles or press 
  releases distributed on other sites can negatively impact
  a site's ranking in search results.

  Note: I wouldn't expect links from press release web 
  sites to benefit your rankings, however.

What's your point here? That you find these guidelines so specific that they don't leave any room for interpretation?

I think the point here is that this is only a complicated topic if you are trying to follow the letter but not the spirit of the guidelines.

So you can't figure out exactly how you can game the system... boo hoo.

> Organic search and paid search are silo'd.

According to who? Google? Is there any transparency or any way for an outsider to verify that?

Yes. According to Google. It is a myth that is years old.

Matt Cutts answers: Do AdWords customers get special treatment in organic search results?

http://www.youtube.com/watch?v=7aV5DmL_eog

If there is evidence that Google gives special treatment to Adwords customers in organic search then this could be verified as a lie. As long as this evidence remains nothing more than accusations and suspicions, then I choose to believe Google that there is no teacup orbiting Venus. It seems unlikely, but not impossible. I am willing to sway my view when I get evidence. Till then, I will treat it as a common newbie myth.

Every single startup I've worked for, one of the objectives was to befriend Matt Cutts so you would have a quick line to getting stuff like this fixed. Seems like a reasonable amount of power to give one person/company...

You will be surprised by the amount of time people take to work seriously on fixing their penalty.

I have helped over 40 websites recover from Google penalty in the last 7-8 months. 80% of the websites seek help only after they have screwed up atleast one reconsideration request without taking the necessary steps.

Google typically responds to penalty reconsideration requests within a week, often less than a week. So if you have taken the right action and have submitted a really solid reconsideration request, its totally possible to get out of a penalty in under 2 weeks. Also, the backlink count was under 200K which is a reasonably manageable.

Rap Genius could have saved some time using tools like scrapebox to perform some of the scraping activities. But thats another story.

I want to give both Rap Genius and Google the benefit of the doubt.

I like to believe that Google checked the link profile again and found a large amount of low-quality links removed. Google would not remove a penalty if the spammy links were still abundant. So that they are out of a manual penalty is justified.

Rap Genius showed good intentions to comply with the guidelines. They went out of their way to clean up their act. These were no doorway pages or elaborate linking schemes with their own servers. It was a Linking Bieber Scheme that is considered greyhat in the industry, and after this, will probably be less popular.

This was a large and visible PR drama. Of course both Rap Genius and Google were on top of things. Google never stated that this was to be a month-long ban. We don´t know how those thousands of other sites were penalized. Maybe the large majority is so spammy/crappy that they simply won´t rank near the top with their spammy links removed, on the merits of their content and audience alone. For sites that are entirely build on manipulating SERPs, then yeah, a Google penalty can seem permanent. To me, that is an ok thing.

About the connections. Sure they help. I don´t see what is wrong though with leveraging your connections when there are millions on the line. Maybe a connection humbly asked: Hey, we effed up, how can we restore trust? This thread makes it sound like the connections pressured Google into removing the penalty. That seems unlikely if we all give the benefit of the doubt.

This was not greyhat. This was blackhat. Paying money for inbound links on keywords that trasnfer page rank (don't include NOFOLLOW) has been blackhat for a LONG time.

It was Panama hat. Rap Genius did not pay money, they offered a tweet. In a crude and obvious enough way to be a little scheme-y. Which unfortunately for RG was picked up in the media.

"even I was scratching my head to figure out if this was actually a Google violation or not. Rap Genius’s apology post had the company deciding itself that maybe it violated guidelines that links should be “editorially placed." - Danny Sullivan.

At least now we know. This is blackhat.

Paying with a Tweet from an account with a large number of followers (which definitely has value) instead of money is still paying. At best, it's basically a link exchange, which is also not allowed.

Is it so inherently wrong to prioritize a high-profile case of a popular site? Merit is not a synonym for equality. From a user perspective that decision might surely have a lot of merit.

BTW, it also helped that people really found Rap Genius useful.

Er, what? You suggesting that Google execs are benchmarking the productivity of google searches? There are a million other "substitutes"...hence the shady SEO...

Google is constantly improving the search results. If factors, such as a low bounce rate and high dwell time, show that a site is popular and useful, why not rank it higher?

People tend to think in binary: A site is either spam or no spam. But spam is much more complex. A site can be 90% good and 10% bad. A site can turn spammy without their intention: comment spam, hacked sites. A site can pull out all the stops to hide their link buying. A site can be useful to have in index, regardless of some spammy behavior.

In order to judge if something is better, it presumes absolute reference points. In otherwords, google ranks the "productivity", generally. And then only later, refines the relative merits within a sub-class. (And if this was true, it would have nothing to do with Rap Genius.)

Avoiding the (questionable) debate on the productive capacity of planet-earth...the "model" of what customers' "really need" is questionable. Do they need (1) information that is factually correct?; and/or (2) do they need the actually meaning or interpretation of (1). And if its true a subset need (2) what is the ratio of (2/1)?

And only then does this framework make sense.

I have mentioned this before in this thread, I'll do it again. I AM really happy for rap genius.

I also view this post from RapGenius as a kind of a PSA.

They used the reasonable doubt argument, and won, they could have kept quiet about it. But, they let people know that others could appeal on reasonable doubt too.

Which is commendable!

Unfortunately, the underlying message some people miss when they feel this is about bashing RapGenius is that it is not.

The outrage is against Google. More specifically about 3-4 things:

1. Google has penalized sites for way lesser, and refused to reconsider, even when reasonable doubt was jarringly obvious (ex. some sites affected by panda/penguin algo updates targeted towards content/blog farms were penalized for as less as having similar posts type of intra-site linking as all content farms were using it)

2. Google will never apply this reasonable doubt argument for anyone in the future, unless they have friends like the investors at Andreessen Horowitz.

3. Google has given no indication on how they modified the algorithm for this special circumstance, the question that people are thinking is more in regards with if this was a team decision, or an individual's special favor.

4. It's a slap on the face for all white hat SEO professionals, webmasters and founders who have been burned by Google's no bullshit policy. Again its not about the policy existing, that's just a fact everyone has to work with. It's because now enforcement is clearly optional dependent on connections.

2] How do you know this is the case? Matt Cutts said they've relisted them quickly because AH slipped Google some cash? Or maybe RapGenius followed their cleanup policy correctly and were relisted naturally after submitting like many other companies a reconsideration request.

3] How do you know this was a special case? Thousands of sites I'm sure re de/relisted every day that you don't hear about (because they aren't being talked about on HN and no one outside here cares), are you sure none of those get relisted as quickly as RapGenius?

> How do you know this is the case?

I don't. I also don't know if this is not the case. Do you? This uncertainty IS the reason for the current loss of faith.

> Matt Cutts said they've relisted them quickly because AH slipped Google some cash?

I never said cash exchanged hands. I doubt it did. Can you with a 100% surety say that a special favor of any kind did not occur?

> Maybe RapGenius followed their cleanup policy correctly

What clean up policy. If there is such a policy why isn't it public, why doesn't anyone know of it? [0]

> (RapGenius was) relisted naturally

A manual modification of the algorithm is not what has been established as natural over the last decade+.

> Thousands of sites I'm sure re de/relisted every day that you don't hear about (because they aren't being talked about on HN and no one outside here cares)

Ah, but know tens of these sites that were penalized, as in know 'em in and out, know the whole team, know of every SEO practice they implemented, these only include those that were penalized for having site structures similar to content farms (themes, permalinks and similar post type cross linking between posts) that were not even replied too let alone getting a chance. I also know a lot of SEO professionals who don't know any site that was afforded this (I emailed 'em, if I hear of one I'll update the post).

These sites were all with 300K+ uniques a month && 1MM+ pageviews/month.

HN is not my only source of information and news.

> are you sure none of those get relisted as quickly as RapGenius

Yes.

edit: [0] the only tool that I know Google allows webmasters to use to reduce potential penalty is the disavow links tool[1]. In the past I have seen this only useful for people who were targeted by link bombing etc. at times it was because a SEO practioner did in fact use some shady link sources.

Kudos to RapGenius for talking about it in their blog post, nonetheless, the results were still too quick (not judging rapgenius or their connections... only targeting Google).

[1] https://support.google.com/webmasters/answer/2648487?hl=en

At the point where you start suggesting that Matt Cutts was bribed [or accepted a favor], you know you've left the land of rationality and entered into wholesale speculation. Your argument is on the face ridiculous, but just to be clear: 1) Matt Cutts is very wealthy and doesn't need any favors. 2) Matt Cutts once penalized Google's own Chome site for improper links. If there were ever any conflict of interest that ever existed, that would be it, yet Matt went ahead with the penalty.

EDIT: added []

> At the point where you start suggesting that Matt Cutts was bribed

When did I do that?

I have not and am not suggesting that Matt Cutts or anyone at Google took a bribe of any kind whatsoever.

I have on the other hand said that there is some uncertainty if favors may have been given/used to expedite the process of re-listing.

Which if true, in itself looks really bad, simply because of the position of authority Google sits at.

Pray tell, when did a favor for a friend, acquaintance or anyone for that matter, regardless of their alleged wealth or social standing come to be defined as bribery/monetary value exchanging hands?

You haven't addressed my argument at all. Replace bribery with favors, and my argument stands.

I used both interchangeably (see point #2), and I've edited for clarification.

If you want to keep going on the overly semantic line of argument, bribery = gifts, not just money.

Favor[0] == Help

Or quite literally if I go by a dictionary: an act of kindness beyond what is due or usual. Example: "I've come to ask you a favor". Synonyms: service, good turn, good deed, kindness, act of kindness, courtesy.

To further elaborate to address your failure to understand a semantic line of argument:

1) The terms Favor and Bribery are not interchangeable.

2) I have only used the word favor.

3) I have insinuated[1] that there is possibility[2] of an alleged[3] favor having occurred in expediting[4] the re-listing of RapGenius, and that this may have happened due to good connections from the VCs.

4) If true, this is not in any way illegal, by the law of the land (as opposed to bribery[6], which is illegal). And this would simply be a moral wrong doing to others who cannot use this speedy service based. This is due to the position of power Google has on web start-ups, websites and businesses.

5) In addressing your previous comment I stated that none of my statements can or should be construed[5] as referring to bribes or any thing of monetary value, this includes gifts.

Meanings and Definitions:

[0] Favor - an act of kindness beyond what is due or usual

[1] Insinuate - suggest or hint (something bad or reprehensible) in an indirect and unpleasant way

[2] Possibility - a thing that may happen or be the case

[3] Allege - claim or assert that someone has done something illegal or wrong, typically without proof that this is the case

[4] Expedite - make (an action or process) happen sooner or be accomplished more quickly

[5] Construe - interpret (a word or action) in a particular way

[6] Bribe - a sum of money or other inducement offered or given in this way

You missed my point again. The semantic line of argument is pointless. Favor and bribery are interchangeable because my argument holds for either case.

Your insinuation is still baseless.

So you're saying they were given a Blueprint to fix their mishaps but got off based on Reasonable Doubt before they went thru with the recommended plan of action?

I suppose it's just a Hard Knock Life for companies that don't have prominent investors backing them. It's almost as if the investors pulled an American Gangster type situation on Google.

Well I suppose at the end of the day this is a reminder to Blackhat SEO users to Watch The Throne.

Off-topic, but

> "To save a drowning man, he must first give you his hand"

is very wrong from anything I ever read about the topic. Drowning leads to involuntary reactions that cannot be controlled, like trying to climb on the person trying to save you, endangering them as well. That's why approaching and grabbing a drowning person from behind is suggested, and maybe even knocking them out if need be. (unless you can throw them a lifesaver or something else they can grab safely, of course)

Ehh, the story is the double standard google is showcasing. The earlier story about Expedia and this outcome tell it well.

This is rapgenius. If it had to cross the line why on something so materially insignificant.

God this pisses me off to no end. So my site is targeted by negative SEO. (Yeah, I know, I know, negative SEO doesn't exist. Fuck off.) So who do I call? I've been penalized for months and there is nothing I can do about it. Nobody can help. But these fuckers are able to make a call and recover from a SELF INFLICTED wound in less than a week. I am being targeted by crap that is COMPLETELY OUT OF MY CONTROL and it's tough luck little buddy. The double standard here is fucking sickening.

Disavow?

I've tried the disavow. Seen no indication that it had any effect.

There were 31 thousand new backlinks added on December 31st (Holy shit, I just noticed that. WTF!). Another 1300 added on the 1st of January. It's normally 100 or 200 a day. Not sure how I'm supposed to manage the disavow process when I get that many new spammy inbound links a day.

maserati problem

Why would you say that? This is impacting my Google search results TODAY. If I turn off Adwords, my leads drop to virtually nothing...

This scares the crap out of me. There are some really shady people in my industry and I'm sure they wouldn't think twice about launching a negative SEO campaign against me. I wish Google and Matt Cutts would address this for real.

This post is complete bullshit. Of course they have relations. Maybe they cleaned their backlinks, but still I cannot believe how they got back so fast in SERP.

Anyway, Franz Kafka would die again if he saw his text on this site (http://poetry.rapgenius.com/Franz-kafka-a-dream-lyrics), so let's just forget about this site and don't use it.

Don't be so spiteful. It might very well be that they have relations, don't make it sound like that's a bad thing. Besides that they also seem to just have cleaned up their act very thoroughly.

And about Franz Kafka, I think he'd turn in his grave if he heard about how petty people like yourself criticize a website that tries to make literary criticism of his work digestible and enjoyable for the masses.

> It might very well be that they have relations, don't make it sound like that's a bad thing.

It is a bad thing here: it shows that Google is apparently willing to act preferentially to favoured businesses, undermining the idea that search results are provided with neutrality.

No it doesn't. It means that Google employees can do stuff manually, when asked by a friend or colleague. It says nothing about Google as a business favouring Rap Genius as a business.

The fact that you can request a google employee in a position to help you out to do so displays enormous leverage.

Imagine a business in Germany, Vietnam, or even.. Texas having to do the same? It's pretty much impossible. This is what happens to everyone else out there.

Which is what the parent is referring to when he says that google results aren't as neutral as they claim to be. They clearly favored someone with leverage in this instance.

But this is the only instance that we know of. So far.

Although I understand your concerns, this is just how the world works. If you know the president of the United States personally, there's a good chance he might spend a little of his good will smoothing out some bureaucratic processes for you. Nothing illegal or abusive, just the president asking official this-and-that why process so-and-so is taking so long for your company.

It's not a question of having to do the same, it just means that knowing a Google employee with some magic power is a valuable asset.

But yes, if you're a company that doesn't have this asset and you make a stupid mistake like Rap Genius did your site might be off Google for a month or two, possibly destroying your business.

I still don't follow how that makes Google not neutral in any significant way.

Okay, I see why you were not able to follow the problem.

>> stupid mistake like Rap Genius did your site might be off Google for a month or two, possibly destroying your business.

Sometimes businesses get wrongly blacklisted. Yes, it happens. They do get destroyed because of this, people do lose homes and jobs over this.

But this isn't because Google takes a month or two to sort this kind of thing out (a month or two is also not really acceptable - too long - given the impact Google has on a business). It's because experience shows that once you are on the wrong side of the big G, there's pretty much nothing you can do to get on the right side. It's no longer a question of a month or two, it's about registering a new domain, and starting from scratch. Sometimes you have to do this even when you didn't do anything wrong. This applies to almost every Google service whether it's Search, Adsense, Gmail, or anything else.

But Rap Genius definitely did something wrong. It was obvious for everyone to see. Yet, they managed to follow the 'recommendations' and get back in the game pretty quick. We know that this is probably because of connections because every business out there that gets blacklisted by Google will make sure that they are 'regulation-compliant' ASAP. Yet, many of them never manage to make it back and those that do take years at times.

Yes, knowing the US President helps, but that's provided you can get your work done without knowing him as well, in a reasonable time frame. Especially when it may make the difference between running a thriving business and being homeless.

QED: Monopolies are bad.

Alright, I fully agree :)

One thing I know is that RapGenius is good about making a big stink, being in the news, and then having things work out in their favor, even if they were the ones doing a dumb thing in the first place.

They freaked out at Heroku with decent reason because it was costing them money, but if they ran their own infrastructure, that would have been a total non issue and at the scale they are running, why ARE they using Heroku?

They did some incredibly dumb SEO things because they are greedy and are willing to cheat to win. That is on RapGenius, but once they got banned it is somehow Google's fault and they should be reincluded? I don't get it.

RapGenius is great at one thing - creating controversy and getting press for it. I guess that works for them, but it seems like a very selfish way to grow a business. Instead of focusing on building value themselves, they are willing to tear down their business partners in public to get what they want.

I'm not sure why this behavior hasn't put them in the same category as Zynga, Groupon, Swoopo, and all the other companies the tech industry loves to hate.

Are you arguing that what Heroku was doing wasn't wrong?

I'm not sure why this behavior hasn't put them in the same category as Zynga, Groupon, Swoopo, and all the other companies the tech industry loves to hate.

Comments on every single RapGenius post on HN have been filled with vitriol toward the founders, their attitude, and the culture they foster.

They have the word 'lyrics' at the end of the slug for that post and a number of others which don't have any relevance to lyrics. Surely that's still bad SEO practice?

Like lyrics of Franz Kafka, did not notice before reading your post.

http://poetry.rapgenius.com/Franz-kafka-le-nouvel-avocat-lyr...

Well spotted, it looks like all of their news article URLs end in '-lyrics', even those when it's not even slightly relevant eg: http://news.rapgenius.com/Snapchat-find-friends-abuse-lyrics

More shady SEO practices from Rap Genius, disappointed but not surprised.

Very observant... I wonder what Google would say about this. I would almost find this more egregious than their original offense due to the posts having nothing to do with lyrics.

All of the links in the poetry section I can see have "lyrics" appended to the URL.

And some of those are not even poems:

http://poetry.rapgenius.com/Herman-melville-bartleby-the-scr...

This one is particularly weird / egregious:

http://poetry.rapgenius.com/Engineering-genius-understanding...

Not really bad SEO practice, just inflexible software.

Tom Lehman here (Rap Genius co-founder)

The fact that all our text and song page URLs end in "-lyrics" is a relic of a time when we hosted almost exclusively lyrics.

Making the changes required to host all kinds of annotated texts has been a big project and making the URLs make sense is something we haven't gotten around to.

The reason it's not a totally trivial change is that both users and texts share the same top-level namespaces, and so right now we use the "-lyrics" suffix to differentiate song and text pages from user profile pages.

We want to change the suffix to "-annotated" for non-music pages and will hopefully get around to in the next few weeks. To our knowledge the existing situation doesn't help with SEO (no one is searching "bartleby the scrivener lyrics"), and it's confusing, so we 100% agree that the current situation is bad.

For more info on the "Worse is Better" philosophy that caused the current jank situation to persist for so long, see this talk: https://www.youtube.com/watch?v=X45YY97FmL4

Its interesting that despite their founders outspokeness, the influx of heavy weight VC, tech press expose's on their offices, etc., RapGenius is really no more or no less than a standard, multi-multi-page SEO site, no different than the countless others just like it. No brand, no user affinity, no direct traffic to speak of. Just a bunch of drift net pages crawled by Google waiting to pick off unsuspecting search traffic. My favorite line of this post was the last: "Much love. iOS app dropping next week!" - unless you are searching that term in google, Im not sure who that is addressed to.

I wonder how long it would take for any other website without 15 mln backing and direct line to Google to return.

As someone who knows folks who got wrongly blacklisted.. Well, you pretty much never get back. No matter what you do.

You're lucky if you get a decent reply from customer support to begin with..

Your best bet is to change domains, fill in a redirect and try to build up backlinks again. It'll take a year or two and hopefully no competitor pops up in your space.

Or, you can try and get Andreesen-Horowitz interested in your business.

I have helped sites with over 2 million backlinks and got them out of manual penalty in under 3 months in very competitive verticals like travel & finance.

It entirely depends upon the scope of penalty. Unless you are completely deindexed from Google search results, you can always recover from a penalty.

Feel free to email me if you have any specific need.

Pretty interesting decision to include a lengthy "technical digression" in their SEO apology:

<< Technical Digression: How to scrape 178k URLs in Ruby in less than 15 minutes - Ok, so you have 178k URLs in a postgres database table. You need to scrape and analyze all of them and write the analysis back to the database. Then, once everything’s done, generate a CSV from the scraped data. >>

As I see it, the "technical digression" serves two purposes:

- first, they use it with Google to show how sincere they were about "cleaning up"

- second, and perhaps more important, it gives Google a public excuse to justify why they restored RG so quickly in rankings. As in, "Oh they did a very thorough job of complying by our standards that we had no option but to restore their status".

Unfortunately in this case Google's response speed comes across as a bad thing. Because it applies to only a very select group of sites that are either very rich (aka large advertisers) or very well connected.

[Edit: clarified imagined comment]

My favorite is that they seemed to have spent a few hours (more than 3!) improving their code so that it would run in 15 minutes instead of three hours.

So which hit should google show for a query like "the fox lyrics"? Of course, it should be a page that contains the lyrics, but which one? The most popular one? The one shown by google will be the most popular one.

From an IR perspective, search for lyrics is somewhat strange. I don't think it's easy to say if some hit is more relevant than the other. Sure, more popular sites may offer better UX (or simply less annoying ads), but at the end of the day it's a search for facts.

I wouldn't be surprised if google will serve those lyrics themselves some day. Freebase (and hence the google knowledge graph) already contains tons and tons of information on pratically any popular song (organized in a confusion way via recordings, canonical versions and whatnot) - artist, genre, album, release dates, length of a recording, etc. I don't think adding the lyrics as well is totally out of question. After all multible KBs are user curated and users provides lyrics.

Just like they serve "fact-hits" from their knowledge graph for queries like "movies by spielberg", google should then be able to serve the lyrics directly. While this is bad for lyrics sites, imho it is convenient for users and follows the same prinicple like "movies by spielberg" or "wife tom cruise"

People should worry why Google has so much power nowadays, rather than why they unban Rap Genius so fast.

This really is the thing that struck me. All this talk about companies being destroyed and jobs lost and hardly a mention of worry about the power Google yields.

We need more competition in this space. Sites need a search engine with real market share to compete against Google.

yup. just adding duck duck go to this laptop now. thanks for the reminder.

https://addons.mozilla.org/en-US/firefox/addon/duckduckgo-ss...

This really shows that even in Silicon Valley, your success is driven as much by the connections you have and the people you know, rather than pure talent and merit.

The "technical digression" with Github code on how to do a mass scrape is cool, though not surprising (er...how else would you accomplish such a big data task...though I'm glad I'm not the only one who uses Typhoeus, a great but underused gem).

But what's surprising to me is that a company that knows how to automate things and work with data...why the f--- were they manually soliciting bloggers to post links? That seems seriously inefficient and comically backwards...as if Sergey Brin spent his early Google days modifying SERPs by editing and uploading Excel spreadsheets to the server.

It seems seriously inefficient to me because at this point, Rap Genius has a lot of traffic and a lot of linkage already. So did the few backlinks, per day, that RapGenius got, through lame blogmasters who would participate in such a thing...really have an effect on their SEO rank, i.e. is it so easy to still game Google?

Or, was RG just doing it because they thought it would have a worthwhile effect (keep in mind that this is one of their co-founders who was individually emailing people), i.e. RG was being naively optimistic?

Looks like this involved quite a bit of work on Rapgenius' part but I am glad they cleaned up their mess and apologized to Google.

Moreover, this article feels pretty neutral/humble and techy, a far cry from the haughty attitude their founders usually display in interviews.

http://m.entrepreneur.com/article/230631

But now, Rap Genius co-founder Ilan Zechory tells us the site has returned to Google's search results. "It takes a couple days for Google to re-index everything, so search results are a little wonky right now, but we are officially reinstated," Zechory said in an email.

Somewhat ironically, Zechory served as a project manager at Google for two years prior to founding Rap Genius in 2009.

>> Wow that is troubling. So site that gets banned gets unbanned quickly and founder just happens to be ex google project manager

I've followed SEO on and off for years and I wouldn't have thought that their strategy of allowing people to generate and embed links back to the site was shady at all. I even recommended something similar a few years ago in good faith.

They're useful, relevant, and it just seems like good marketing. These things will spread organically and drive traffic even if you remove the SEO benefits.

Could you say that Scribd or Slideshare are doing something similar by allowing people to embed documents in their websites along with a no follow link back to their site?

"nofollow" is the key.

Allowing people to create links to your website isn't shady.

Paying people to link back to your website on high-value keywords like 'justin beiber lyrics' from blog posts on other sites to artificially inflate your pagerank is 100% blackhat SEO. The payment in this case was a Twitter post about the link provider's website.

Clear as day there is a double standard. That's what should be discussed.

Actually, it's understandable that Google set this on High Priority...

#1: it's a high traffic website, that is a startup and the whole deal got a lot of attention of us (HN Readers) + Matt even saw this here. So it was good to respond fast (no choice)

#2: RapGenius is dependant on Google (although they try to let you see it as a social network, but it's not), they would have gone bankrupt ( i suppose) without Google.

#3: Without a quick response and/or measure to fix their shit on their end, investors wouldn't invest any more money in RapGenius and Google would probably hurt a lot of future investments (how is this startup dependent on Google, ...)

BMW once got a penalty from Google, they didn't got easily off. But their business wasn't dependent on Google.

Either way, there are both pro and cons to each action. But overall, i think they handled if fairly well.

I am sure they went through many sleepless nights during the holidays. Hella sucks. I think Google is doing the right thing and I am sure not just related to seo that rap genius learned a lot about themselves and will become stronger... as well will the overall tech community.

Disavow tool actions take months to bring any effect(if any).

after a year I still see "unnatural" inbound links in webmaster tool. So I would say it have no effect.

Interesting closing paragraph:

>Though Google is an extremely important part of helping people discover and navigate Rap Genius, we hope that this ordeal will make fans see that Rap Genius is more than a Google-access-only website. The only way to fully appreciate and benefit from Rap Genius is to sign up for an account and use Rap Genius – not as a substitute for Wikipedia or lyrics sites, but as a social network where curious and intelligent people gather to socialize and engage in close reading of text.

ie, you should really just become a member rather than relying on google...

Also, where was bing in all of this?!? ;)

Other search sites dropped the ball on this one

what does not make sense to me is that after removing all the back-linked urls, should not that drop Rap Genius's overall ranking even lower than before, since the artificial boosting has been removed. Why they are back on the top. googling "lyrics", RP is number #6 in the first page. I think RP has received special treatment here. Google search team needs to clarify this.

Well atleast now AH partners can get back to posting rap lyrics on twitter. We wouldn't want anything to stop that bit of entertainment.

The tech part is obvsiouly here to calm down people from Hacker News and make them say "Oh this part was good, after all they are not so bad". It's just a shitty scraper but they talk about it like it was a revolutionnary script...

There is no way they could get back so fast and so easily on google.

The SEO strats / growth hacks weren't entirely misguided. Rap Genius just didn't perfect their activity and didn't let it exist entirely behind closed doors. If Rap Genius wanted to risk it and engage in blackhat-like activity, it should have at least kept things more quiet and less spammy. Rap Genius should have kept the finer details, at least, to a phone call with the blogger.

Google can't see a local business handing their customers/client a $25 gift certificate in-person for a random positive Google+ review. Along the same thread, Google would also not be able to see Rap Genius flying out a community outreach rep to meet with a prolific music blogger at a bar to go over an exchange for a positive mention and authoritative link. Just saying.

Were the 178k URLs that they scraped on 178k unique domains or a very few domains?

The tool they wrote looks like it would have been a DoS attack if the latter were the case. There appears to be no thought or consideration for the sites they scraped that had linked to them, no attempt to read robots.txt and see whether they should fetch the pages.

What they did was wrong in the first place, but they could try and do things better in their attempt to repair things. Instead they continue to act in a way that externalises any cost to them and continues to treat their service as somehow privileged enough to deserve such breaks.

How many of the 53k failures to pull and scrape pages were to do with the fact that the servers in question were under a DoS attack by RapGenius?

Probably the story is missing the help of a highly skilled SEO consultant/company but it's impressive anyway. To get it done so quickly they had to work 24/7. Kudos!

I have to note, you have to be big to make comeback like this. Google provided disavow tool, but from my experience it simply not working. So it is manual action for sure.

Infinite scroll (or some type of weird JS) won't let me get to the bottom of the page... Ain't got ish to do with this but I just thought that I should mention.

If Google were to have a year long ban on RG stuff to leave a precedent, Would people start to view Google as a sub-par search engine because Bing returns "high quality" results like RG still?

It's also rather weird how a lot of people seem to view Google as this public service for the people by the people etc, it's not. They can do whatever the hell they want and if you don't like it, simply stop using their service to send a message.

I think it will take time for them to regain their rankings for non-branded keywords.

I just searched for justin Bieber lyrics and they are sitting in middle of page 3 on google.

They are ranking for branded searches, but not money keywords.

Although long term, I see this being a net positive for them due to the thousands of backlinks they got from all the press around this story.

I see it taking them several months to regain rankings...especially if the links they disavowed were artificially propping them up.

While we're on this topic, are GroupOn's API Branding Requirements in violation of Google's webmaster policies?

http://www.groupon.com/pages/api-branding-requirements

They make it a requirement of using the API that "nofollow" is not used: "Don’t ‘nofollow’ your links to Groupon."

Is it a sign of getting older when black background web sites get harder and harder to look at for longer periods of time?

I'm waiting for their next SEO single. "Oops, I did it again!"

Snapchat could learn a thing or two of how to do PR from this post.

Rap Genius appears to have offset their drop in search results traffic with all the publicity they've been getting from being penalized by Google.

I give them a lot of credit for being transparent with their learning mistake, taking ownership, and describing what they did to fix it.

"tis better to apologize later, than ask for permission first."

This is a relief. RG certainly gives the best search results for a lot of queries.

This whole story made me fear that Google was choosing power over user experience.

And that should never be the case, no matter how offending the founders of RG might appear sometimes. Don't be evil!

Is there a smell of corruption? Because it is in favor of power, connections instead of user voting, just like in favor of dictation over democracy. Isn't it?

Fair voters = valid pages linking to RapGenius

Bought voters = manipulated SEO spam blog posts

Fair voters >> bought voters

Not reacting to the SEO manipulation would have meant accepting a fraction of bought voters.

Banning RG completely would have meant to ignore the even bigger fraction of valid voters.

Google's actual decision seems to be the most democratic: Forcing RG to decrease the number of bought voters while not dictating over the voice of the valid voters.

You have an interesting point. But it's hard to say:

Fair voters always greater greater than bought voters. When you buy backlinks from the SEO consulting service, they gave you hundreds or thousands everyday, just like some people complained in the comments.

"Google's actual decision seems to be the most democratic: Forcing RG to decrease the number of bought voters while not dictating over the voice of the valid voters."

The problem is: how can they tell which is which?

I doubt this type of criteria is reliable. I collected some information from the user comments along with other research and wrote a blog post here:

http://bingobo.info/blog/bingobo/what-are-the-current-seo-pr...

Maybe you will also take a look at the possible solutions here:

http://bingobo.info/blog/bingobo/how-bingobo-can-help-busine...

Money wins every time.

Certainly a double standard exists at Google. I originally thought finally Google is taking a firm stance, but this reversal only does harm to Google's image imho. Take home message is, blackhat SEO is forgiven if you are funded by connected people. Don't have funding or connections but make the slightest infraction in SEO? Feel the google girth.

Reply on news.ycombinator.com