Example: Why your Title to URL algorithm shouldn't chop off partial words... | Hacker News Reader

Example: Why your Title to URL algorithm shouldn't chop off partial words...

bbc.co.uk

101 pointsspxdcz24 comments

Not only funny, but bad SEO. They'd be better off separating the words rather than concatenating them.

Funny, but more suited to reddit.

That's an interesting comment. I'm still trying to 'work out' the Hacker News crowd (even though I've been here almost two years!).

I've submitted plenty of things that get to #1, or front page, but can't seem to find much consistency in what people like / don't like. Not as obviously as Reddit/Digg/Slashdot, anyway - which is a good thing! I'm very pleased that HN isn't as one-dimensional as some other sites; it's what keeps me coming back.

But it is weird, what becomes popular and what doesn't. For example, I submitted this particular story, which I was a bit unsure about (like you say, more suited to Reddit), but still people vote it up.

Other times, I can submit content that I think is genuinely interesting/fascinating, that is much more technical and comprehensive than this rather silly BBC/URL story, and it doesn't get a single vote. Maybe it's the time of day / day of the week.

On that note, does anyone know of any data repositories for Hacker News front page items? An API or raw-data download that can be analyzed? I remember someone a few months ago doing some analysis on the best time of day to submit, but was wondering if there was any public data out there, or whether I should start spidering/collecting my own?

An API? That's a sad joke. (I say that bitterly as someone who's been knee-deep in the HTML structure of HN for the past few days, trying to parse out data for my own app.)

I talked to someone in #startups a little while ago who was developing a server-side API for HN, but last I heard it wasn't done.

There's a few sites that are sort of "HN aggregators." Unfortunately I've lost the links, but there was one that used a simple algorithm to accumulate the top stories of HN. It would scan the front page every N minutes and give each story X points based on its ranking at the time. Does anyone remember that one?

At any rate, the best you could get from those sites would be a friendlier HTML structure that doesn't feel like 1994. You'll probably have to parse some HTML yourself.

Yes. A "hurr hurr" link is now the top HN link. A sad day.

Don't complain, flag.

This is what makes you sad?

In a way it's disappointing. I view HN as an island surrounded by a sea of tl;dr and we're fighting to keep it off our shores.

i think he means it more along the lines of the "unfortunate" meaning of the word.

and it is, kind of. it doesn't seem to fall in line with HNers expectations of articles. there isn't any content. it is just a "lol look at this" submission, which doesn't typically float to the top here because of the nature of the community, but will at reddit.

To be totally honest, I'd rather have HN allow posts that are of dubious topicality than to turn into another stackoverflow where topics are closed/deleted if they don't meet the strictest definition of what's allowed on the site. Sometimes the harm of off-topic posts is much less than the harm in doing away with them.

Then flag the link.

indeed. this is a cross-post.

URL normalization/canonicalization is already a hard work but chopping off URL to avoid dirty words in any language looks also difficult. How could we implement that?

First, we need to know the existing dirty or vulgar words in a specific language (and maybe a region). Do we have WordNet for that? in any language? with a classification, a dirty word is not the same as an insult but sometime an insult can be a dirty word. I don't know any good reference of that. There are some websites with insults but a good dictionary...

Now if we look at the algorithm to do so, you will also need to know the language used in a page. It's often fine to have a French speaking page including a chopped off URL containing "cum" but not for English.

Wait a minute? Is this a problem really important? At the end, it's the only way to publish a recipe on HN...

A simple enough solution would be to simply not include partial words. They don't help SEO, and when your phrase is long enough to require it, it's unlikely you'll have uniqueness issues.

Isn't this an example of why it should chop off partial words?

This is evocative of ferrethandjobs.com, although in that case it was a matter of capital letters not coming through.

URLs seem to require their own grammatical rules to avoid outrageous results, and this is just considering English; incorporating other languages likely to be in one's target audience would be important, too. One saving grace is that most languages don't sound much like each other... most of the time. (Which makes when they do all the more painful.)

Beware foreign borrowings, too...