Found my notes on difference in differences method. First model the sessions originating on page [i] at time [t] as:

    S[it] = b0 + b1 * X[it] + b2 * A[i] + b3 * D[it] + e

Where:

    S[it]: number of sessions originating on page [i] at time [t]
    b0: base traffic for all pages
    A[i]: base traffic for page [i]
    X[it]: seasonal traffic for page [i] at time [t]
    D[it]: dummy variable, 1 for after treatment 0 for control and before treatment
    b3: effect size of treatment
    e: noise/error term

Taking the average over time:

    E[S[i]] = b0 + b1 * E[X[i]] + b2 * A[i] + b3 * E[D[i]] + e

Take the average from S[it] (the first "differences"):

    S[it] - E[S[i]] = (b0 - b0) + b1(X[it] - E[X[i]]) + b2(A[i] - A[i]) + b3(D[it] - E[D[i]]) + e

which simplifies to:

     S[it] - E[S[i]] = b1(X[it] - E[X[i]]) + b3(D[it] - E[D[i]]) + e

Split all pages [i] in to control and treatment groups 0 and 1. Take the "difference in differences" of the two groups:

    (S[1t] - E[S[1]]) - (S[0t] - E[S[0]]) = b1(X[1t] - E[X[1]] - X[0t] + E[X[0]]) + b3(D[1t] - E[D[1]] - D[0t] + E[D[0]]) + e

Given D[0t] = E[D[0]] = 0, and for a large number of pages X[it] - E[X[i]] = X[jt] - E[X[j]]:

    (S[1t] - E[S[1]]) - (S[0t] - E[S[0]]) = b3(D[1t] - E[D[1]]) + e

Which gives a formula in the form of y = mx from which we can calculate the effect size b3 and the p value using linear regression.

The key assumption is that X[it] - E[X[i]] = X[jt] - E[X[j]] for large numbers of pages. I found this assumption does hold in practice. I'm not a statistician so forgive me if I've made any statistical errors but I think this analysis is correct.

I might write up a blog post at some point to explain it in more detail.