Hacker News Reader

Comment by mturmon | Hacker News Reader

Of course, for many web sites, the events are not independent.

If I visit one page, I'm likely to visit several more. If I see an interesting page, I'm likely to forward it to friends. If I happen to have 1M twitter followers, my visit could spark a cascade of visits.

Unless your models of site visits are very good (good enough to verify independence), you would need to examine logs to determine if effects like those above kick you completely out of a domain where independence is a good approximation.

Note, time-dependence (a non-constant but deterministic rate λ(t)), is usually easy to cope with. Lack of independence is much harder, because then you are left with Cox processes (http://en.wikipedia.org/wiki/Cox_process) and worse.

View on HN

Those are good points. The event in question is not a simple user visit, however. Let's say it's a particular user action that's a bit more involved, e.g. filling out a form. The assumption here is that users fill out this form independently from one another.

Ah, I just noticed you are the OP.

Sure, you have to start somewhere with any analysis, and Poisson is the place to start. Anyone (like me) can then question independence assumptions from the sidelines.

I was moved to speak up originally because a commenter referred to inspecting the logs to verify the Poisson assumption, and in my experience, looking at the data is always an excellent principle, and generally preferable to just stopping at an intuitive understanding of the arrivals process.

Correct me if I'm wrong, but surely in the scenario that you are Ashton Kutcher and tweet a link to your followers, the average inter-arrival time would decrease but overall it would still be a Poisson process?

It would not be Poisson, because the arrivals are not statistically independent.

Let's take a simpler case, where I have one follower who has a high probability of clicking links from my tweets, and there is a webpage that only I know about. I visit that page, tweet it, and an hour later, with high probability, my follower visits it. The event

  { I visit page }

and the event

  { follower visits page }

are not independent, and neither is the arrival process associated with it.

In the extreme case, for large times, there are either zero visits to the link, or two, but never just one (obviously non-Poisson).

The Kutcher example is just this, magnified.

Independence is a very strong assumption, but it is often casually stated. Independence is what allows you to decompose a very complex process (all those arrivals) into their individual atomic events (a single arrival).

Without invoking independence, you have a very complex task in relating how each of the N possible sources of events might relate to each other. It could be arbitrarily complex (e.g., some followers might only click a link if two people they're following like it, unless Ashton likes it, in which case they will never visit it, but if Colbert likes it, that will override Ashton).

Thanks for taking the time to explain!

I understand that each { follower visits page } is dependent on the { Ashton Kutcher tweets } event, but I still don't understand why events are not independent of each other, and why the N visits to the webpage in time period T from the tweet would not likely be described by a Poisson process. Not expecting an answer btw, just thinking out loud here. :)

Reply on news.ycombinator.com