Understanding Waiting Times Between Events in a Poisson Process
nbviewer.ipython.org
nbviewer.ipython.org
If I visit one page, I'm likely to visit several more. If I see an interesting page, I'm likely to forward it to friends. If I happen to have 1M twitter followers, my visit could spark a cascade of visits.
Unless your models of site visits are very good (good enough to verify independence), you would need to examine logs to determine if effects like those above kick you completely out of a domain where independence is a good approximation.
Note, time-dependence (a non-constant but deterministic rate λ(t)), is usually easy to cope with. Lack of independence is much harder, because then you are left with Cox processes (http://en.wikipedia.org/wiki/Cox_process) and worse.
Sure, you have to start somewhere with any analysis, and Poisson is the place to start. Anyone (like me) can then question independence assumptions from the sidelines.
I was moved to speak up originally because a commenter referred to inspecting the logs to verify the Poisson assumption, and in my experience, looking at the data is always an excellent principle, and generally preferable to just stopping at an intuitive understanding of the arrivals process.
Let's take a simpler case, where I have one follower who has a high probability of clicking links from my tweets, and there is a webpage that only I know about. I visit that page, tweet it, and an hour later, with high probability, my follower visits it. The event
{ I visit page }
{ follower visits page }
In the extreme case, for large times, there are either zero visits to the link, or two, but never just one (obviously non-Poisson).
The Kutcher example is just this, magnified.
Independence is a very strong assumption, but it is often casually stated. Independence is what allows you to decompose a very complex process (all those arrivals) into their individual atomic events (a single arrival).
Without invoking independence, you have a very complex task in relating how each of the N possible sources of events might relate to each other. It could be arbitrarily complex (e.g., some followers might only click a link if two people they're following like it, unless Ashton likes it, in which case they will never visit it, but if Colbert likes it, that will override Ashton).
I understand that each { follower visits page } is dependent on the { Ashton Kutcher tweets } event, but I still don't understand why events are not independent of each other, and why the N visits to the webpage in time period T from the tweet would not likely be described by a Poisson process. Not expecting an answer btw, just thinking out loud here. :)
Verifying that the process was homogenous involved analyzing the historical data. I didn't include it in the writeup, but that was done with a likelihood ratio test.