edhallen

I co-founded a company that found itself with a similar set of criteria for what we wanted. For us, VC worked out well. We weren't the typical company they saw, but going in with a very clear set of things we wanted from a partner was very helpful. Ultimately, they were high value add when relevant but otherwise let us focus on growing the business.

My guess is it comes down to the right VC/Partner and finding mutually agreeable terms.

I think it probably is, it just takes more planning. Take email A/B testing - if email is a main driver of your usage activity, you could split your users or list into two groups and then send them different emails over a period of months. It'd be slightly more manual (you'd have to segment the list) and then leave one group off for a few months, but it would let you easily gather very significant data.

These are great points, and speak to why measuring the impact of email is really, really important. Until you're holding your emails accountable to performance (whether it's getting someone to come back or to buy), you're just throwing darts in the air and potentially annoying someone.

At Klaviyo (I'm one of the co-founders, so note bias), we've helped many web apps and ecommerce sites setup full email strategies - and we definitely find that some emails don't work. You've got to test different emails and strategies, see what works - and then ideally optimize and personalize what (or even whether) you are sending based on individuals' actions.

Having now spent a lot of time sending and analyzing email campaigns/triggered emails (I'm one of the co-founders of Klaviyo), we keep seeing that email is one of the best ways to turn analysis into action - not least because it gives you a complete feedback loop.

For example, say an Ecommerce store realizes most people only make 1 purchase and don't come back. With email, you can target exactly those people, but you also can quickly see if it worked (because you know who you sent it to, you can see if they actually made a purchase).

Retention is definitely really important, but I think what's at root here is that email is an ideal way to interact with users in a more targeted and personal way outside of when they proactively visit your website. The same feedback loop idea should (and will) apply to push notifications, texts, in-app messaging, etc as time goes on.

I'm setting up Inky now and excited to try it.

On the note of smart views, any idea how representative my data is of the typical email user?

Separately, I'm curious to know how far smart views go with aggregation - i.e. if I get 10 facebook friend request emails, are they combined? Or is this an irrelevant question in the smart view paradigm?

This is a really interesting point - I think the question will really come down to how well campaigns / businesses can measure the impact of their marketing. If your messages/posts aren't driving donations or purchases (or hurt the long term usefulness of the person you're marketing to), then you won't run them.

The problem is that today most marketing is blind - completely unlinkedin from the impact it causes. I'm counting on technology / big data / analytics to change that.

This point about Google and Facebook is interesting, not least because Google has certainly had numerous people join the campaigns. My guess would actually be that the level of testing by the campaigns (just based on the small subset of data that PP has) is incredible and much, much better than either of the tech giants - regarding email.

As a benchmark, the most analytically savvy paper mailers / marketers of the last 20 years could arguably be Capital One. They did elaborate testing that involved not running TV ads in markets for years at a time, tons of different message styles, frequencies, etc - all because optimizing these results was worth so much to them.

In short, if getting email right means millions of dollars, then there's a premium on getting it right - just as there is for Google for nailing search testing.

I'm working to get it back up, but it looks like our host will keep us down for 30 minutes or an hour. Good sign that we need to invest in better hosting.

It's a great call - I'll post a follow-up with that detail and try to find a couple other examples from people in other geographies and demographics. My take (from this exercise and looking at ProPublica's data) is that behavior varies significantly based on both WHO campaigns think you are and WHAT you've done lately (what you've read, which sites you've visited, have you donated or volunteered, etc).

This complexity is going to make analysis nearly impossible in the future as political (and marketing) messaging becomes incredibly personalized.

(I'm actually author and OP here is my also-HN reading brother who beat me to the punch)

It's good to see Mixpanel building out its people analytics offering, but more broadly good to see momentum in the trend towards measuring the impact of email and other communications on customer behavior. To this point in time, to many companies send emails in a spray and pray fashion - not knowing if they really impacted customer behavior on feature usage, purchases, etc.

We're working on the some similar problems at Klaviyo (customer lifecycle management / targeting customers and measuring the impact). One of the most interesting related things we've seen is that measuring the impact of emails goes along way towards eliminating debate about email frequency or whether to send campaigns - because you can always just send the campaign to a subset and actually know if it works.

From Patrick's course to articles like these, it's great to see the growth in interest in this. The challenge with LTV (as with all analysis) is making sure that you can translate it into action. For many companies, the barrier is just getting started.

On a broader note, it's great to see how many companies are tackling customer lifecycle management (though with very different approaches and different end markets). It's time people stopped building one-offs for this.

A few companies I know of (the first of which I'm a co-founder of):

- Klaviyo

- Vero

- Customer.io

- Reamaze

- Intercom

- Totango

- Mixpanel

I think you make a great point here about the confusion that often gets put out there between data and analysis - a confusion which I'd say is implicit in the term big data as well (and hence I ran with - caveat, I'm the author).

As far as the problem with the term "intelligent data" - I think what you say is exactly true if you do data analysis one time; however, the issue is that for those of us running startups, we find ourselves doing analysis over and over - so intelligently selecting data (in a way that takes us less time together and leads to the same decisions) is a huge win. Read intelligent data as being data + intelligence - not a new type of data.

Likewise, the problem with asserting that more data is always better ignores how most companies are making decisions. At the end of the day, our analysis is completely meaningless without a new action. So a better analysis that doesn't get implemented is worth far less (nothing) than an analysis that gets implemented successfully and drives results.

Helpful feedback. For most of our current clients, we find that the amount of value we generate / cost of what we replace (lots of engineering work and excel usage) is a lot higher than this.

That said, I think our sense is that for startups / newer firms it would be great to come up with some sort of plan that makes this more feasible. We'd love to hear thoughts and we're certainly glad to figure out something that works based on specific needs of people. Email me at ed.hallen@klaviyo.com and glad to chat.

Never thought of the Klout point, but it's a good way for us to think about it.

Good question. Our current focus has been on high-margin businesses with relatively few (hundreds to tens of thousands, not millions) customers, primarily because they seem to really feel this problem.

Integrations is both through 3rd parties (for things like Mailchimp, Zendesk, etc) where it just takes a couple of clicks and no development and through our API - we have both a javascript and HTTP API. Our javascript snippet gives us a logins per user, but you can really pass us anything you want (has this user finished setup, have they used feature x, etc)

This stuff is hard to get right. I think the answer for most web apps is to shift more towards trigger based emails, and then to measure how particular rules change customer behavior over the next 24 hours, week, 30 days, etc. Rather than an "A/B" test per se, you really need to split customers into 2 groups and intentionally not send an additional email to a group.

Good look at the history of the marketing term big data. I think this fits squarely in the realm of big data hype, or at least a review of why it is big data hype (something I've blogged about here http://www.klaviyo.com/blog/2012/07/16/the-curse-analytics-b...). The SAS discussion makes it especially clear that it's viewed as a trend.

But - posts like these frequently jump to the top of hacker news, and there's clearly a feeling that access to new analyses and more data is going to change the world. I think this is probably right. I'd love to hear more discussion of how big data has already started this process.

Examples I can think of (and I'd love to hear more): - weather analysis for farmers (weatherbill / the climate corporation) - marketing (the target pregnancy example - http://www.nytimes.com/2012/02/19/magazine/shopping-habits.h...)

More examples?

This article gets a few things right, but I find it largely to be another example of playing up the hype without talking about the real meat of big data's potential.

Some of the more useful points: 1.) Big data's promise has led to massive advances in technology, and companies are spending billions on this new technology 2.) The limiting factor with Big Data is analytical in nature, not necessarily in storage, processor speed, etc.

What this article fails to address is why big data is so valuable. While it gave one example of a real use case (identifying influencers in social networks), if anything this served to highlight the problem with Big Data. Are companies completely shifting their marketing budgets to target influencers? Is this driving major financial impact for Facebook?

I have no doubt that companies are actively identifying incredibly powerful new uses of big data, but for big data to be truly revolutionary, we need to develop the analytic methodologies and decision-making processes to benefit. Reading the McKinsey report cited in the article is illuminating. There's a ton of value cited, but it isn't particularly clear how companies changed their decisions based on big data in a way that drove the value.

(I posted more thoughts on this broken link between big data articles and actual decision-making here: http://www.klaviyo.com/blog/2012/07/16/the-curse-analytics-b...)

This is a great addition to the unfortunately too limited category of articles about how big data / analysis changed actual decision-making.

That said, the definition of big data used by this article doesn't strike me as what I would think of as big data. Based on what it says, the dairy industry was changed by analysis and data collection but nothing that couldn't be stored on your typical phone. The article hints at big data (via greater genetic analysis) transforming the dairy industry in the future, but the massive changes in cow DNA so far are seemingly due to "small" data.

This confusion of big data with just solid analysis and decision-making happens a lot, but does a good job of highlighting how much progress there is to be made in using data to drive decisions (independent of how much data we use).

Great post - if anything, I think this problem goes way beyond just real-time dashboards to extend to the vast majority of analysis done today (both by web and more traditional companies).

The statement that particularly resonated for me was [figure out] "if you’re looking at stats now because you’re curious and impatient, or because those stats will actually drive business decisions", but I'd take it one step further.

Analysis and stats are incredibly valuable when:

1) they are applied to a real business decision that is tied to actual value

2) you are willing to change what you are doing based on the result

3) you don't already know the answer

4) you have sufficient confidence in the result to act on it.

We need to get more used to stopping and thinking before we start analysis by laying out the decision we need help making, the different paths we're willing to take, and the amount of confidence we'll need to change our decisions. I could definitely be a lot more disciplined about it.

(I actually wrote a blog post last week on this same topic that lays out the above criteria in more detail and might be useful - though that was a reaction to the profusion of ads out there calling Big Data and Analytics "hot", that ignore how they actually drive real value. Blog post is here: http://www.klaviyo.com/blog/2012/07/16/the-curse-analytics-b...)

I think this distinction between personal and impersonal emails is a good one. The frustration (for me at least) comes in when email isn't relevant, when it's redundant, or when it's unnecessary. My guess is that the problem isn't actually email itself - but it's instead the systems that keep it impersonal.

To paraphrase a blog entry I recently posted (http://blog.klaviyo.com/2012/06/18/making-emails-better-pers...), we need better informed, more intelligent systems and thought to drive what emails are sent/received. For instance:

- There's no reason a company should email me about a feature I already use or a product I already buy. They know this - they just haven't bothered to integrate their systems.

- There's no reason I should have to send 10 emails to schedule a meeting where/when everyone can make it. Our calendars should be able to make this easy and painless.

- There's no reason I should get 4 separate emails from friends containing the same article. Can't I get one message saying four people sent you this?

As Seth Godin said in his Ted talk, something's broken. All of these examples I cite can be solved - and there's a ton more just like them.

As stated, it's crucial that privacy be fully respected for users. The key thing may not necessarily be knowing exactly who someone is, but instead knowing what they've done (which features they use and how often, which marketing emails they open, which support tickets they file, etc) and using this to give users better experiences personalized around their history of interactions. Providing this type of experience from web companies is what we're working on at Klaviyo (http://www.klaviyo.com).

In most cases, companies are tracking all of this data, just in multiple different systems and not bothering to pull it together (i.e. why do I get emails about product features I already use?) to use to make my life better.

On privacy, companies need to make sure they are being open with users. For most of these so-called "people analytics" companies can choose whether to include personally identifiable info. Companies need to be intentional, and should choose to anonymize customer data when they can (but should still treat people uniquely based on their past interactions, even if they can't put a name on someone).

As the comments show, there's a clear need for this that lots of great folks are going after (Vero, Intercom, etc) - but it all treads the fine line between usefulness and annoying for the end customer. If I look at some of the key examples on the Vero site, some seem to fit the obviously non-annoying bucket (email people if they haven't logged in in two weeks) while others are more dubious (send a user an email when they buy ten widgets).

Two solutions to this: - Doing some meaningful analysis that compares successes/failures within the same cohorts to identify really important inflection points (i.e. a user needs to complete setup within 48 hours or they never will)

- Talking to customers to really understand the onboarding process so that you can use these event triggers to provide even better customer service.

These tools are definitely better than nothing, but I'm left thinking that we (as a community online) might be able to do better. Is the answer to better customer service on the web really more sophisticated automated emails? Or is it actual personalization + meaningful analysis?

Great post on the usefulness of cohort analysis (or of experimentation more broadly, which if you think about it, is exactly what cohort analysis is - it just uses the past as a control).

One thought on ways to analyze the follow-on problem of customers canceling after 61 days (a problem similar to what I've seen at every web company I've ever worked at).

First, perform the same cohort analysis you’ve already done, but look at the cancelling customers vs retained customers at day 1, day 15, day 30 and day 45, then use this analysis to figure out your triggers (things like # of Facebook posts needed by day 15, % of profiles claimed by day 30, etc).

Once you have your triggers, you can make proactively calling / emailing problematic customers a key part of your daily routine. While discounts might still be the way to go, this trigger based approach is one I've seen work well. Additionally, because you are in touch with problematic customers it often gives you insight into what do next.