nehagetschoice

Yep - the AI-generated copies in the first run are pretty generic - that's not the value add (sorry if the demo made it appear that way). The key value adds are (1) Quickly running an A/B test with no code (2) Running inferences based on the test data and feeding it back to the model to generate better copies in the next iteration(shown in the demo video)

+1 that's my understanding as well.

This is cool. We definitely spent weeks sitting in customer interviews, then parsing information and patterns from those into meaningful insights, and sometimes that's just so hard to do, that it wouldn't even be concrete enough to fundamentally change our roadmap. I'm excited to see if this tool can bring nuanced insights to research conversations.

We either run experiments with higher traffic diverted to treatment (eg: 50% instead of 1%), or run it for a longer time. This is the reason why we need companies to have at least 2K monnthly users to run pilots with us.

So far, we have seen 1) Companies like stat sig that are stand alone AB test platforms 2) Customer engagement platforms like customer.io that allow you to do A/B testing on their campaigns 3) There a few website only platforms like coframe and mutiny that do not integrate with internal products like notifications and logged in pages

The things are make us stand out -

1) There is no tool afaik that looks at inferential learning based on past experiment results on copies. The experiment analysis and the continuous feedback loop is missing. 2) To make (1) happen, the first step is copy versioning and management. Without a tool that can abstract strings in a CRM, and monitor learnings over time, it's hard to make (1) possible. 3) We integrate with tier 0 services like notifications with a low latency solution that makes copy iterations possible for in-product features, outside of logged out surfaces like web pages.

Would be interested to know if you have seen software that solves for (1), (2) and (3)?

Interesting. One of our customers just requested this today, where they want to test form fields. Are you imagining higher form submission conversions by optimizing the number, type, and layout of the fields?

Yeah - two themes here -

1) Is copy even important? I think it is. If this post was titled, "Auto-tune experimentation for short-form content optimization", it might make half the audience confused about the product. In fact, the 1-liner we use for HN is very different from when we talk to VCs, because the audience is different with different goals & backgrounds. I guess the point I am making is that messaging has to be contextualized, depending on users, platform, and goals.

2) PMF vs copy - I agree that the two are orthogonal. Copy is not going to solve for the lack of a PMF (and it shouldn't). Exactly the point above - the goal is to help more and more users comprehend what you do, hopefully in a way that's more personalized to them.

Appreciate the candid comments and opinions here. I'll break it down and go over them - 1. Having access to data is not the problem we're after (most companies have the first-party data in-house). The key challenges are around having a platform that fundamentally separates strings (copies) from code, and lets you update them effortlessly, based on inferences from that data. So, I am not sure I understand why this is a Google/Meta product? 2. UX is not the value add from this product - agree with you on it (even if it appeared to be the emphasis). The ability to make scientific edits without re-deployments and accelerating continuous iterations based on user feedback, is what we are going after. 3. Curious why you think A/B test results are fictional? Getting stat sig results is probably the surest way to conclude results. Perhaps there's a different angle you are talking about here? 4. RE: don't A/B test at all. Given the number of users that get exposed to every change a consumer company as large as Twitter makes, not testing can be disastrous, which brings up another great point - Large companies are struggling to use all the (generic) gen AI content today, because it needs to be performance tested before it can be placed in front of millions of users, and that's not scalable today. 5. You may be alluding to another good point - copy is as much as art, as it is science, and writing it well takes context, quality, and expertise. That's something we hold a strong opinion on, and we don't see this or any other tool eliminating that expertise. The goal is very much to streamline and augment those workflows.

I'm curious to learn what kind of challenges you had in your past growth roles. It would be great to understand specific examples/workflows, and how you dealt with them.

Yes, we think of this as a decision tree, where initially, copies may look different by say, user demographics and location. As we learn more about what's working well for different dimensions of users (eg: topical interests, traffic source, platform type), the decision tree grows, and every single element in the anatomy of copies is optimized based on past learnings. In an ideal world, every user truly sees a unique copy tailored to them.

You'd be surprised how few companies actually do any systematic copy optimization - it remains ad hoc even at the big players, which is one of the reasons we started this startup.

In speaking to SMBs and large companies, our insight was that the problem of copy optimization resonates more with larger companies, as smaller companies are more focused on survival/basic marketing techniques like opening up new channels. Larger companies have already exhausted those levers, and are ready for more sophisticated optimizations.

Another reason (branding) why copy would look different for different products :)

The idea transpired from some of the early tools at Twitter. The actual product has nothing in it that would be considered Twitter's IP afaik. A couple of other companies we spoke to use similar techniques to optimize copy testing. Good call out on the language though - we could have worded it better!

Love the idea of doing demos on landing pages that can more easily benefit from copy optimization. We will build the next one for Intuit!