illdave

If you look at http://wap.bing.com/robots.txt, the URLs that Google is returning are actually all set to 'allow', not disallow.

It also looks like m.bing.com/robots.txt blocks /search while their actual URLs are /Search - I guess Googlebot treats robots.txt as case-sensitive.

There's a bit of a weird myth with robots.txt and the idea that it prevents pages from showing up in search engine indexes. Robots.txt means that the search engine cannot crawl the page - it can still include it in the search results if it sees enough sites linking to it. It can take a guess at what the page title can be, but there's usually no descriptive snippet because it's unable to see what's on the page.

If you don't want the page to be included in the index at all, you can use the meta noindex tag, by putting this in the head:

<meta name="robots" content="noindex" />

Pro tip: you need to also unblock that page from robots.txt - if Google isn't allowed to crawl the page, it can't see the meta noindex tag, which means it would stay indexed.

Thanks, I'll add that in the next update.

You can sign up and ask for help to sharpen your skills :) - you wont be listed to help anyone until you click 'help a fellow hacker'

Thanks very much - those are really good suggestions, I really appreciate it - I'll work on them both for v2.

Clickable: http://hackerbuddy.com

I guess they'll put the streaming video up when it starts (10am PST, so in just under 2 hours). That seemed like the most appropriate page to link to.

Clickable: http://bigthink.com/series/62

From my experience, Googlebot doesn't crawl pages that are blocked in robots.txt files. Check out Bing's robots.txt: http://bing.com/robots.txt - notice how /search is disallowed. That typically means that Googlebot isn't able to access that page. The same for the other search engines, it's more down to if they specify (through robots.txt) that Googlebot isn't allowed to crawl those results.

I don't think that's true - at least, I've not seen any evidence. There's a good chance the Google toolbar sends back browser data, but I think it's extraordinarily unlikely that Google snoops on what people are searching for on Bing, and what results they end up clicking on, and then adjusts their search results on the back of that.

I've heard some bad things about 99designs too. While I've not done this, it may be worth looking through dribbble.com to find a few designers that have done something you like, and get in touch with them. They're generally excellent designers - I have no idea how much they'd charge, but could be worth asking.

Wow, this is really impressive!

I wonder if the money is withheld because it'll be returned to the advertisers - or whether Google actually keeps it. I'd be surprised if Google charged the advertisers for traffic that it detected to be fraudulent.

I would love to know whether Flickr is profitable. It's obviously an awesome service, but I imagine the running costs must be pretty high.

From an SEO point of view, I completely agree with solost - using subfolders like techinch.com/apps is much stronger than subdirectories or individual sites - especially if the topics are all relatively connected.

I'm not entirely sure how easy it is to do and it would take a bit of hacking with your Wordpress template, but as a compromise you could look at having each category page (so /apps, /widgets, /whatever) have it's own background image - although I imagine completely different themes or very different styles could be a bit of a nightmare.

It's really, really difficult to know who to accept SEO advice from if you're not sure of what needs to be done. The best advice I can give is to first learn the basics yourself, read Google's latest guide: http://www.google.com/webmasters/docs/search-engine-optimiza...

I'd also recommend reading some of the guides on seomoz.org - you don't necessarily need to know enough to be able to handle it all yourself, and create your own search marketing strategy, but knowing the basics will help you enormously if you choose the route of pitching for help from an agency or freelancer.

I would generally avoid 'automating' SEO - stay away from software that makes unrealistic promises. There are some tools that can help give basic advice by grading a site, but you should definitely stay clear of software that promises to improve your rankings all on it's own, or software that says it will automatically build a ton of backlinks to your site (9 times out of 10, that software will hurt you in the long run).

If you do choose to eventually hire an SEO, or an SEO agency, you can generally tell the good ones from the bad because they'll be happy to answer any questions you have. Good SEOs will be open and honest, will NOT guarantee to get your site ranking for your selected keywords and WILL sit down with you and run you through a proposed strategy for improving your site, finding the right keywords and working out the most effective way to build good quality links to your site. I'd also recommend asking them for case studies that can demonstrate effective results for previous or current clients.

Good luck!

That's just extraordinary.

Lloyds TSB in the UK, too.

That's weird - possibly because this listing was for http://thewildernessdowntown.com while the first was for http://www.thewildernessdowntown.com (note the missing www. subdomain).

It seems pretty cool, but I get the feeling that it'll actually be more of a hindrance in reality. It's much quicker to type a password than it is to let it scan your face. I love the idea of it, but I don't think I'd ever really use it.

I think that's a genuinely awesome idea - and I totally agree with hgarg, consider using Facebook ads to test the water. Apparantly you can get a free $50 coupon with new accounts http://www.netpaths.net/blog/50-facebook-coupon-advertising-... (although I've not seen that, I don't remember getting one when I signed up - I could easily be wrong).

On his point about Google placing the news for "payday loans" at the bottom of the screen instead of the top - I really do think that's algorithmic, rather than a concious decision by Google to hide bad press. In the UK, I'm seeing the news box appear at the bottom of the results for "car insurance".

This is probably the smallest nitpick ever, but thought you'd like to know there's a typo in the header - "Guaranteed minumum" instead of minimum.