No Emails on Saturdays
swapped.tumblr.com
swapped.tumblr.com
A couple of things I forgot to mention in the post.
1. The numbers are based on hits through URLs with tracking information. Some visited and voted from the Twitter feed, some hit the base URL. This affects absolute numbers, obviously, but the %-ages should be about the same.
2. I also tallied up negative and positive votes, thought that perhaps people on Monday were more stressed and negative than on Sunday. But there were too few dislikes, so it's hard to make any conclusions -
like !like
Sat 22 3
Sun 44 3
Mon 34 1
Tue 34 1
Wed 35 4
http://www.thumbtack.com/labs/abba/#Sunday=47%2C389&Tues...
If we treat "Sunday" as the baseline and use "votes" as the count of a successful email, it appears that you can say with a somewhat high degree of confidence that Sunday is better than Saturday, but nothing else.
I've always been a fan of sending between 3-6am, although I think I just took this as a rule of thumb from someone I worked with an never questioned it much. Would love to see a similar study on send times within the optimal range of days.
"Most emails are sent between Monday and Friday, with the highest volume on Tuesday and Thursday. More subscribers open email, however, on Wednesday and Thursday."
"Subscribers are likely to open email after 12pm, and the most active hours are 2-5pm."
"Your newest subscribers are your best subscribers. They’re most engaged immediately after signup, and then trail off over time."
Those are averages though, your mileage may vary. For the newsletters I send out, Tuesday morning has proven to be the sweet spot.
More information: http://kb.mailchimp.com/article/when-is-the-best-time-to-sen...
So you are saying that if I had 10x the data, then the Saturday opening rate and the Sunday response rate could've been completely different?
Perhaps something was happening that Saturday, something that the post failed to take into account. Perhaps something happened during those weekdays, causing more engagement than normal.
Interestingly enough, nothing was happening on that Saturday. That I actually checked.
I obviously don't disagree that there's a lot of variables in play and some vary faster than others. What I meant by "10x" was that if the list were 20k subscribers instead of 2k for this particular experiment - what would've been the chance of the Saturday open rate jumping up.
It doesn't invalidate what you've found, but does mean you should now continue on with your experiment to increase the sample size - and help ensure your data wasn't impacted by something else - e.g. vacations / a conference that was attended by many of your subscribers / etc etc.
To answer your specific question, if you had 10x more data, your margin of error (with 95% confidence) would drop to 1.39%, so yes, you could reasonably get fairly different results.
I was disappointed to read that you skipped Thursday and Friday. In my domain, Mondays are the peak of activity, slowly sliding down to a bottom on Friday/Saturday. I would have liked to see your numbers for every day. Next time! (In that case, your MoE would have been 5.37%, btw.)
PS. I don't know why I picked (mod 5). Makes zero sense now.
Expected range: 36.2 (average of the clicks) P-value: 0.14
Typically you look for p-values of less than .05 to indicate statistical significance. So unfortunately this one test suggests the data falls within the range of normal (random) distribution.
[edit: maybe i'm not understanding, but engagement on sunday is highest at 47, yet the final recommendation is for "Tuesday or Wednesday" (35 and 39). why wouldn't you choose the day with highest engagement?]
[also, incidentally, those numbers are all the same to within a couple of standard deviations (1 sd ~ sqrt(n) ~ 6). so the sample size is too small to draw any really firm conclusions.]
> Not on Sundays or Mondays, but if pressed, pick Sundays.
Seems consistent to me...?
It's actually reasonable. This is an opt-in mailing list and subscribers are generally interested in the project, so if they are hit with a relevant update on their free time, they will respond.
(edit) The rationale behind "Tue/Wed" is that I value views more than votes. Agreed on std.dev., yet Saturday still stands out as the least performing day.
If I understand the data correctly, Sunday has a low open rate, but of those that open, it had very high engagement rate.
The author seems to be saying that the open rate is low enough to make Sunday a "bad day", but that really depends on what your email is about.
If getting people to read the email is your priority, then avoid Sunday, but if you mostly care about getting them to respond in some way, then Sunday could be good - those that read it will have the free time to actually respond.
(Of course, while the limited data provided by this experiment is interesting, it's probably not worth basing any serious decisions on)
The exact pattern will depend on the audience. If it's VmWare emailing IT people, then the open rate is probably going to be the same through the workdays and zero on weekends. If it's some hobby stuff, then it'd be in reverse. For my particular project, it just happens that Saturday sucked, which is a good thing to know.
- If nothing else, it shows that there is a difference, and the day you pick for your newsletter matters. - It seems that it would be helpful for other people to repeat your experiment (but stretch it to 7 days) to see what the results are for their readers. - The difference between the open rate and engagement rate is quite interesting.