Some are obvious - Yahoo vs. Google, Emacs vs vi, etc. But if I'm making a "website that does X", and X is pretty narrow, how do I find the competition? (Beating them is no problem... :-)
Before we did this, I did a lot of competitive research.
I searched on all the relevant keywords. I ended up finding a ton of comic book retails, plus a lot of other related sites that had excellent ideas (collection management, social networking, pull list management, etc., etc., etc.). So: right there, by digging deep into the search results, I found a lot of things that I didn't even know existed, and this let me expand my concept of what we wanted to deliver, and it also let me get a much better handle on what the market is like.
After identifying the big names in the space, I went off to Alexa and other sites, and measured the traffic of each of these competitors.
I also got some information via backchannels on the likely revenue of 2 or so of the 10 big competitors, and then verified it through another backchannel. I then did some curve fitting to come up with an algorithm that predicts the revenue of an online comic book store, based on their Alexa rank and traffic.
I then went through the top 10 or 20 sites, and took notes. After I saw the commonalities between them, I created a spreadsheet, and listed all of the salient details of each competitor.
At this point I had a good handle on (a) what price point we wanted to aim for during our growth stage; (b) what features we might want to offer.
Then I set up a survey on Wufoo (I ran the survey past a survey designer first to get her imprimatur on it), and sent email to the top 10 or 20 comic bloggers. Two or three of them linked to the survey, and we ended up getting around 500 people to take the survey.
I wrote a script that imported the CSV data from Wufoo into our database, and then wrote a bit of Ruby code that did a mediocre version of cluster analysis. This let me find out which groups of customers would be most profitable, and in turn let me find out what features these customers would be interested in.
Then we wrote up some design documents and started coding.
We coded for three months, and went live in a "quiet beta" two weeks ago. We've been hunting bugs, and will announce to our list of 500 or so survey takers on Monday or so.
That's how I do market research.
-- Travis Corcoran, president HeavyInk.com
"Your comics are here."
there are more cost effective ways of seeing what the market thinks besides disappearing into a hole and coding.
good hackers get leverage and sustainable competitive advantage by solving hard problems, or at least creating software that is difficult to write. saying it's not worth building anything that takes time doesn't make any sense.
And secondly, good hackers can be hired too; it is possible to replicate most things.
I think that valuable web entrepreneurs are, in general, good hackers and good businessmen. A lot of hackers think the business end isn't important and a lot of businessmen think the technology isn't important. They are wrong. You will probably have serious competition and you need to be prepared to compete.
This isn't a kind industry. A lot of teams get crushed. Your team ought to be able to compete on both the business and technical fronts.
edit: I noticed you are one of the founders of DropBox. I don't know a great deal about your product. But, as far as I can tell we agree. DropBox seems like a straightforward, simple, focused application.
If you have direct competitors then it won't be too hard to tell who they are, you do some basic searches and you will find them.
But I think you should look at indirect competition as well... look at all those companies that if they intend to come in they can crush you over night (and I don't mean Google, Yahoo and the other big guys necessarily, I mean all those other companies that their business models allows them to do the same thing as you do, if they like to do so, but they never thought about it or had the intention to do it... sometimes they might have some strengths that would make them more successful than you.)
Also, I would keep in mind that if your product differentiates from the competition by just a few features you don't have a huge advantage
And if you ever intend to compete with any business that has any social aspects to it (i.e. web 2.0 businesses) never forget the network effects they could have just for being out there before you
then search, look on forums, go to trade shows and ask potential customers what they do to solve the problem you want to fix and how they would find you.