Ask HN: I built a site that just went ridiculously viral. What do I do now? | Hacker News Reader

Ask HN: I built a site that just went ridiculously viral. What do I do now?

370 pointsmarkbao188 comments

Hey HN,

So I built http://threewords.me - literally an MVP that I posted on Facebook and my friends started using. Fast forward 3 days. The entire Twitter results page for "threewords.me" is of tweets that happened less than five minutes ago. The site grew 2x in pageviews over the past hour.

Two problems: 1) what do I do now? 2) how do I afford this?

Advertising? Hosting partner? ...daresay... investment?

Sorry to be brief. If you're curious, the stack is Rails + Ruby Enterprise Edition + Passenger + nginx, which isn't cheap like PHP to host. On the $40 Linode right now but maxing out CPU at 350%.

227K pageviews today. 50K uniques. 71% traffic referred. 8,285 users. 3,100 new users in the past 1 hour.

What happens now?

EDIT: Hello! This thread is not going unnoticed while I take my metaphorical fire extinguisher to the fires that are happening. Will reply soon.

EDIT: cranked the Linode up to 4096. $160 server, woo!

EDIT: David from Duostack (http://duostack.com) is helping with the load on his cloud Ruby platform. Many many thanks to him.

View on HN

No offense, but putting it on HN in the middle of traffic spike pressures would not help things. ;)

I was able to sign up and see just enough to get a feel before the site became unresponsive. I like that it's dead simple. I can also see how it's incredibly viral as it asks you to ask others to describe you -- very smart. At first glance it looks to me like something that could enjoy tons of traffic, but which would probably be short-lived. The challenge would be finding a way to have that traffic stick around...

I view this as a bit of a long shot, but I would do a few things. First, you've got to have the site handle the traffic. No advice on how, but find a way to get that done. Next, I would try to become known as the place to get a quick summary on anybody -- even celebrities. I imagine people will receive multiple adjectives from different people, but I'd have the site tally the three most popular and promote those as best describing of the user. :) I see a couple of possible pages for "viewer stickiness". The homepage could feature very popular users. Imagine showing the three most submitted words to describe /paulg or /marrington and other users with large responses... Next, on each user's page it could show their "friends" and the words that describe them. The site also provides the most recently submitted three words on the user's page. This might at least make each user curious about checking back at their own profile to see how people were labeling them. As for monetizing, that's probably with ads, as usual, but you have to be careful about when and where to put them. Like I said, probably a long shot, but who knows? Good luck!

>>Next, I would try to become known as the place to get a quick summary on anybody -- even celebrities.

You mean like About.me is attempting to do? This is an interesting twist though...

However - the quick summary I look for on anyone of interest (professionally) is linkedin.

I do not and shall not have a facebook account.

Yes, but I'd view this one as more more social/playful, not really serious. :)

A gossipy feel might be better anyway. Imagine the "word status" on Paris Hilton going from "hot" to "classy", "generous" and "caring" or something like that... ;)

Good idea. Like a trends.google for celebs.

>As for monetizing, that's probably with ads, as usual, but you have to be careful about when and where to put them.

How about analytics? For each person who writes about me, I could pay to see where in the world they lived, how long it took them to type in the three words (or even watch a movie of them typing them in), whether they clicked the link that was on my Facebook or the one on my Twitter feed (based on HTTP referrer), etc.

If you were confident you could give each user the ability to see this information for 1 user for free, and then have them pay if they wanted to see it for the rest of the users.

Hmm, that's interesting. Very popular users might find some use for that kind of thing. For example, up and coming music artists trying to build a following and connect with fans maybe... I definitely think sites should consider charging when and where possible, but sometimes there is good opportunity for it, and sometimes not.

I recently built a Facebook application that approached the same kind of growth.

Sites like this typically see eCPM between $0.10 and $0.30 for advertising. So you're looking at around $30/day revenue right now.

Consider switching to EC2. This will allow you to easily scale up to a more powerful server if growth continues or down to a less powerful one if traffic tapers off.

Depending on how you set things up, a common bottleneck for Facebook applications is API calls. Ensure you are not making any API calls from Rails, since this will cause Rails processes to hang until Facebook returns results. With high traffic this could easily cause your server to become overwhelmed. Move all API calls to Delayed Job or another background processing system if you haven't already.

It looks like you have relatively few remotely loaded assets, which is great. You can likely speed things up more by making sure that your database is well optimized for the server (Make sure query cache is enabled, and you have indexes on the right columns for example), and identify any actions that take a long time to return.

As for what to expect in the future, growth will not be infinite. There are a finite number of people on earth that are going to love your site and Facebook integration is a fantastic way to reach nearly all of those people in a very short amount of time. After a while most of those people will get bored and you will see traffic begin to fall. The best ways to address this are by consistently releasing new features to reengage users or by referring your users to other similar sites that you build.

Would it be possible to see your Facebook app? I'm curious to see what traits these fast growing apps have in common in person.

2 words: self-interest.

Sorry. It got banned from platform for a few reasons including tagging photos that didn't have people in them. (Similar to the cartoons that people upload and tag their friends as characters).

In general the biggest trait is that the app is built around sharing. Apps where sharing is a secondary feature generally do much worse in comparison to those where the whole point of the app is to share something. Even successful games on the Facebook platform universally require users to share content with friends to progress.

Many of the most basic viral apps are simply based around sending gifts to friends. If 10% of users that receive some sort of virtual gift then go on to send it to an average of 11 friends, your app will be highly viral in no time.

That's really interesting. My recent viral app, Profile Banner, also utilizes tagging of photos without people. How long did it take before your app was banned?

New Feature Idea if it's not already in the works: threewords.me places (foursquare?). would leverage current users engagement to tag locations with a dead simple description. could keep users coming back, provide utility(value)to businesses, and make you some more money. email me (mw63214@gmail.com) if you want more ideas, although it seems you have your hands full as it is. Good Luck!

Also, I don't know how time-intensive it would be, but it could be helpful to have some sort of drop-down box that initiates when users start typing in the three word spaces. Sort of like a google-instant-type thing. 5 synonyms pop up ranging from strong to weak. So if I started typing "h-o-", it would infer I was typing "hot" and (on the left, or strong side) it would have "blazing" and on the right(or weak side) it would have "warm". Or something of that nature. You get the picture. Might be easier for mobile folks to get the best possible description and faster.

hey Mark, if you need server capacity I can set you up with four machines on very short notice, let me know. j@ww.com (no charge, help you handle the spike and see if it has staying power without having to get in to long term contracts).

^ This is reason why I love this place and never believe the occasional "hacker news is going to shit" posts.

To be fair, Reddit has a noted capacity for charity and generosity, but I do agree.

>To be fair, Reddit has a noted capacity for charity and generosity

Understatement of the year.

>Understatement

Entirely necessary here, even a hint of hyperbole and the pedants come out of the woodwork.

Entirely necessary?

This starts the Pink Panther theme playing in my head.

Jacques, you are, and continue to be, one of my favorite people on Hacker News. Thank you. Shooting you an email. :)

Where is your current bottleneck? mysql? check your slow query log, turn on the log-queries-not-using-indexes, do a quick analysis of those to see if you have queries that aren't using an index or queries that can be optimized. Probably 90% of the scaling problems are database issues.

The quick solution is to scale up your current linode package, however, I think you might find that your traffic is going to peak and will wane next week after everyone has gone back to school/work and is out of the holiday mindset at which point you can scale back.

Put in a throttle - /proc/loadavg

Limit it to X signups per hour, asking them to check back next hour. Collect email and notify when they can sign up. Point them at a facebook page, ask them to like the page, do announcements when the next 'batch' of users is able to sign up.

If you want some help analyzing the slow query logs and/or the normal logs, email me a url of a .gz/.bz2 of a reasonable snippet at hnusername@hnusername.com.

Since you're running nginx, consider doing an alias and use proxy caching. I would doubt your static files are causing too many issues, but, if you can cache computationally expensive pages a little, that can help.

Last week I optimized some MySQL queries which triggered the 'not-using-index' warning, even when I had all fields which were used for selecting and sorting indexed. I think a JOIN on a nested subselect somehow confused MySQL: when I duplicated some extra restraining WHERE clauses into the subselect, response shot up from 800msec to 12msec per query. Very nice. But the query still shows up as 'not-using-index', which is strange.

MySQL's query plan is odd at times. What does an explain look like on that query? If any portion of the query, including subselects isn't able to use an index, it will report it. You can try to force indicies with USE INDEX/FORCE INDEX which can sometimes help.

Thank you for these tips. I'll try them out later today.

Ah, I was having trouble with another slow query; however, the EXPLAIN showed that the correct indices were used. I managed to speed it up by restricting the size of the inner SELECT with some extra constraints. From 4 seconds downto 0.1 seconds!

http://cd34.com/blog/scalability/mysql-query-optimization/ http://cd34.com/blog/scalability/mysql-5-1s-query-optimizer/

Very nice improvements. If you haven't already gone through these multiple times, I literally soaked in these last night:

http://www.slideshare.net/techdude/how-to-kill-mysql-perform...

http://forge.mysql.com/wiki/Top10SQLPerformanceTips

Good sets of additional tips/things to keep an eye out for.

Sometimes when you have complex queries it can make sense to create a temp table, create indexes on that, and then query off of that. But queries like that should show up in reporting, not in serving live web pages.

That said, if you're selecting more than 5% or so of the rows in a table, you're probably better off not using an index.

Another random tip. There is a world of difference between putting indexes on 2 fields, and putting a composite index on 2 fields. If you have restrictive conditions on 2 fields, MySQL has to work with the entire list of rows that match at least one of those conditions. (I don't know whether MySQL has the trick of joining indexes together before looking at rows, but even if it does it has to work with a list of rows matching the condition.) By contrast with a composite index MySQL can only look at rows that match both conditions. Much more efficient.

Your site seems to be down.

Think about moving the code over to Heroku now, and keep scaling to meet demand.

If you don't have the cash, email me and I'll front you what you need to keep it up until you figure out how you want to roll with this. This is not a time to trip over pennies.

Congratulations.

Might not be a time to trip over pennies, but there's also no reason to rush into things. Heroku has always seemed to be horribly overpriced as a scaling solution.

A more traditional approach of 1 web and 1 db server (possibly even physical boxes) might end up providing better value and give him enough time to figure out how to scale this horizontally on EC2 (the web part should be easy...remove any in memory state data, the db might be a bit more work).

The release of Hackety was fairly large, I got LifeHackered, TUAWd... 50k uniques, 100k pageviews over three days. Heroku's free account didn't blink.

Sorry but that's nowhere near "huge" in terms of server load.

Even a small machine can trivially handle 150k pageviews per hour, even with ruby strapped around its neck.

Spiky traffic is an issue of its own, but it sounds like Mark's site is growing fairly steadily (albeit steeply).

I agree with latch that heroku becomes (ridiculously) overpriced the further you depart from the free plan. If a stopgap is urgently needed then it may of course still be a valid choice. But in the midterm, if you're pushing serious traffic, look elsewhere.

I'm just saying that it was more than I expected out of a totally free account, that's all.

I have yet to use them at higher loads. We shall see.

Also, it's my understanding that they're still less expensive than something like EngineYard, and still generally cheaper than paying for a sysadmin. Dedicated hardware will of course outperform any of this stuff, but I also don't have to think about it at all. Even VPSes involve spending initial setup time and then periods of upkeep, you can't just install and say "done."

Then again, I'm coming from a mostly theoretical standpoint. I don't have real world experience at true scale.

Yea sorry if I came across rude, didn't mean to attack you or heroku, just wanted to set the figures straight.

With regard to heroku you have a point about the "not needing a sysadmin" part. That, however, only works for a fairly short period during the lifecycle of a business; heroku is ideal for bootstrapping.

Once your site grows to the scale that mark is seeing (if that persists, which I doubt) then the heroku value proposition rapidly shrinks.

At that point you start to need more customization than heroku can provide [for a reasonable price] and you also need at least one person with systems knowledge in your company to prevent expensive mistakes in the software architecture.

Gladly that person will then more or less pay for itself, simply by moving the app to a cheaper hosting platform.

No worries, absolutely no offense taken. :)

EDIT: HN doesn't render that code very nicely, here's a pastie: http://pastie.org/1416631

Hi there, your home page is very simple, render it in plain HTML and post the signup to queue.php.

In queue.php put:

<?php $data = serialize($_POST); mysql_connect('localhost','dbuser','dbpass'); mysql_select_db('dbname'); mysql_query('INSERT INTO cache(cache_data) VALUES(\''.mysql_real_escape_string($data).'\')'); echo 'YOUR REAL PAGE SHOULD BE HERE WITH A MESSAGE ABOUT BEING IN THE QUEUE'; ?>

The cache table should just be a PK cache_id and a text field (or perhaps mediumtext). I've suggested using a database instead of a filesystem based cache because, if you're doing this quickly, doing it using MySQL means you're less likely to run into file permissions or security problems.

Then you're at least capturing everyone's information whilst you figure out how to scale.

You've since mentioned that you've gone for a 4 gig box with 55 Passenger workers. Definitely a good start.

If I were in your shoes, these three items would be next on my emergency-scaling checklist:

1. Database sanity check: Are you using indexes for calls like "Find user by username"? Does your database need its own dedicated box?

2. Pick up another box for memcached. Full-page cache everything you can with rack-cache. Don't worry about expiration, just use some reasonably low TTL to put a throttle on your read requests for now.

3. Move all of your static assets to S3 + CloudFront. Gzipped, with a far-future Expires header. While you're in there, move the JS to the footer of your layouts. No sense serving any of that stuff from your Linode boxes if they're as pegged as they sound from your other comments.

As to making money? No clue. Maybe I'll have an idea when I can get a request through and see what the app is all about ;)

One thought occurs: this seems to be a very vanity-driven kind of app. Find a way to charge $1, $2 or even $5 for some simple premium vanity-stoking feature and you could be in good shape. Better than advertising, in my book.

Have fun!

1) In general, the database is always the bottleneck. It's easy to throw up more webservers- can't do that with db's.

2) Throw up an ad or two, with the goal of having the ad revenue pay your server costs. This will allow you to not worry about the financial situation while you're fighting fires, so you can defer thinking about investment etc until later.

(now after you have the servers tamed...)

3) Make sure you are incorporated. You probably want to be an S corp or LLC.

4) Look at your metrics and decide whether you have a sustainable property or not. (Look at your retention numbers) Based on the data, decide whether you have

(a) a self-sustaining cash cow, (b) something that can be huge and take over the world, or (c) a fad that will grow and die quickly.

If (b), and you need money to get there, start talking to investors. But don't take money too hastily- it's a long term commitment and you'd better know what you're getting yourself into.

If (c), try to sell it to someone who thinks it's a (b) before it dies.

Congrats and good luck!

Not so sure about incorporating, but that's because I spent my day writing up notes on Camz's talk on incorporation. He gave the three reasons to incorporate:

1. You have significant profits or net income. 2. You have high liability. 3. You're getting funding.

I doubt this meets any of those criteria, unless someone offered to fund him.

Here are the relevant links if you want to see what I'm talking about: HN Story - http://news.ycombinator.com/item?id=2047818 Podcast - http://cameronkeng.com/podcast-ep-1-incorporation/ Summary/Notes - http://pastebin.ca/2033868

One thing he doesn't mention is how much "significate profits" is.. Would 1k be significant? 10k?

For example, how much would I save by incorporating if I make 1k versus 10k versus 100k a year?

I'd say it has a high chance of being c. At least where die == level off/decline.

I see a lot of parallels to formspring, whose pagivews/month graph looks like this: http://www.quantcast.com/formspring.me

So you either want to sell as it nears the that peak, or you have to be ready to stick out the trough of sorrow and invest a lot of time into finding a way to grow and monetize it.

So if you have some great vision for how you can turn this into something awesome, super-popular, and monetizable and you want to invest the time in it, you should go for it.

But if not, you're probably going to get some very good offers over the next few days--especially compared to how little time you've invested. If I were you and wasn't totally in love with the site, I'd get out while the offers are good and invest that money into your other businesses. Then when your next site goes viral, you'll just have to wonder how to get it back up ASAP, not how to afford doing so :)

Agreed that it looks a lot like formspring.

But according to Crunchbase, formspring raised a $10MM series B just over a month ago (well after the decline was obvious). I wouldn't be surprised if some of that went directly into the founders' pockets.

Possibly. And if not there's a good chance that they'll do ok after an exit anyway. But they've already spent a year of their lives doing Formspring fulltime, and they'll probably spend a few more years. So Mark needs to decide whether he wants threewords.me to become a major part of his life for several years. If he does, he should keep it. But if not, now's a much better time to sell than after months of putting it on the back burner and having the viral traffic subside.

You can scale up databases, set up mysql replication, most of the queries on a page will be reads, just have multiple read only slave machines.

http://markbao.com/about

You're 17/18?

Kids these days. Geeze.

Fine work sir, though I don't have much advice to offer beyond what has already been stated.

Yeah, and I met Mark 3 or 4 years ago and he had already started at least one and maybe 2 successful startups.

According to http://juniorbiz.com/interview-mark-bao (February this year) Mark has started 11 businesses and 3 foundations, and sold 3 of them already.

Well, it's a nifty idea and I know I've seen that trend on Facebook before. But that's also the problem - it's a 1 time use fad type thing. I don't know how much I'd invest in the current incarnation - perhaps just enough to keep the site functioning. Aim for some kind of traffic-based business strategy like ads or something. Maybe it can help pay for the site itself. If you could perhaps take the current idea and evolve it into something with a little more sustaining power then perhaps you could turn it into a nice little side income thing.

let the words become tags linked to the FB account. Such that if users have a tag like "Snowboarder" associated with their FB account - he can build out a tagDB of FB accounts.

This way - you could target ads to FB accounts/users that are all "snowboarders"

Or you can at least search who among your friends have the tag [snowboarder].

Or if you give more categories to the tags, you can ask your friends to fill in their favorite 3 [X] -- so you can simply ask all your friends a question and then build an interesting DB of tags/info/interests of all FB users.

You mean like this?

http://www.facebook.com/pages/Snowboarding/107496599279538 1,784,641 people like this 9 Friends Like This

The following until you no longer have issues handling the traffic (which might be right after step 2).

1. Upgrade your linode to something around 2 Gig for now.

2. Start playing with your MySQL or Postgresql config (assuming you're not already using redis or something similar, in which case just throw more RAM at the box). Google performance settings for your DB of choice and look for relevant posts to the memory you have.

3. If there's anything cachable, install memcached and implement it.

4. Purchase a second linode and move your database to it.

5. Upgrade each VPS as needed.

6. At this point you get into fancy scaling and there are loads of options. You should have plenty of breathing room by this point though and can plan it out.

Upgrading to the 4 gig linode as we speak. this is getting kind of ridiculous.

The linode guys are great. You might want to flip them an email and let them know you're getting hit. Could speed up the resizing etc.

This has happened to me a few times and usually the Twitter/social media storm blows over fairly quickly. I suggest getting an extra Linode for now and wait a few days to see what happens. If the numbers do stay up the site will probably be able to pay for itself with ads. Your site is down at the moment so I can't really see what it's about.

Edit: Google Cache got it: http://webcache.googleusercontent.com/search?sclient=psy&...

There isn't a twitter/social media storm ie- hype cycle. It's genuine growth with no press or even optimization for sharing built in. It can certainly die down, but it's doing it in a highly organic fashion.

Rather than get another Linode, for the very short term, I'd just crank the current Linode as big as they go (I assume Linode has super-quick resizing) and tell passenger to run a bunch more processes.

Just cranked it to 2048MB. Cranked Passenger to 55 workers in the pool. Using 400% CPU still. Linode's probably not happy.

EDIT: Cranked to 4096

I doubt they're unhappy. Resources are shared in such a way that everyone is guaranteed their fair share. I think the 400% reflects that you have access to multiple cores. 100%=maxing out one core, I believe. (And you might not be truly maxing it out, just maxing out your share of it.)

If you want to see how much CPU is being denied to you because other XEN VMs on the box are using it, in top, look at "steal" - that's the percentage of the CPU that you want that you are not getting because it's been allocated to other people.

Xen is pretty great about partitioning everything except for I/O. Network I/O probably won't be a problem for you (xen doesn't really start getting tripped up until you are well above 1G/sec. I /believe/ linode has a 10,000 packet per second limit on top of that, but that's a whole lot of packets. This is probably not your problem.) but disk I/O will be.

Cache everything. If you cache enough, and have enough ram, slow disk I/O may not be a problem at all. watch your linode to make sure you don't go over quota on your disk I/O allocation... maybe email support to make sure you are in the clear. My understanding is that linode has better disk I/O than I do because they have a process in place for limiting heavy disk users. (my understanding is that it's a good process, they warn you before limiting you and generally do it in a fair manner. I plan on emulating them in this regard. )

No two ways around it; I/O to spinning disk and virtualization go together poorly. What would have been a sequential access on a dedicated server becomes random access... and on a good day, that means 1/10th the performance.

If you are using a lot of disk I/O, a dedicated server is about your only option. Considering the prices nearly all American providers charge, if you plan on keeping it for more than 2 months, you are nearly always better off buying. (and that way you can buy better disk, too.)

Of course, if you aren't limited by I/O, or if you can solve the I/O problem using ram caches, then it might make sense to stay 'in the cloud' on linode or ec2 or the like until you are sure you are going to need the space long-term.

Adsense. With only about 10k hits a day, my bedtime calculator http://sleepyti.me brings in $10-$20/day. At the rate you're getting users, you could probably bring in a significant load of cash. Good luck!

Hey Shaw, it sounds interesting. Can u guide me about how u started this project and contacted Adsense. I wanna start one like this and make some earning.

Depends on the niche and the website. I have made a blogspot blog and brought it to 4% CTR and $20 CPM. That's +$200 a day.

those are good stats. Care to share the URL?

"Sleepyti.me is provided as a free service thanks to support from our ads."

I don't know if you read the "Sacked by a Google Algorithm" article from yesterday. http://www.duckworksmagazine.com/11/columns/guest/winter/ind... The comment you make on the results page sounds dangerously close to what got Adsense to boot him.

Thanks for the tip. I saw that article on HN the other day, but the wording is designed to encourage people to leave adblock disabled to see relevant, sleep-related ads. You might be right though, it's always better safe than sorry; I'll look into clarifying the wording there. Wouldn't want to offend my Google overlords!

Curious: What kind of traffic sources are you getting? I notice an FB like button, is that the sole source?

Facebook and Twitter help spread the message to new users, as well as a nice daily referral from StumbleUpon.

A few people have posted my URL to social news sites like Reddit and Digg, but even when those gain traction it's usually just a few thousand and quite short-lived.

Advice from a guy who's gone through this a few times:

1. Monitor your retention. Are people coming back or just checking your site out once? You can go viral very quickly but tank just as quickly once the initial interest dies down.

2. Start thinking about a business model. Highly viral sites that don't retain well could really hurt your bank account if you're not monetizing. If you can't can come up with an interesting one, throw up some ads to tide you over.

3. As soon as possible, get your database on dedicated hardware. Your bottleneck is going to be the database: so memcache, add the right db indexes (get a backup db so you can hot-swap while you do this), and add lots of RAM if needed. You don't want to be scrambling to migrate your database once you hit the limit.

I don't suppose this traffic will sustain. Slap an ad right in the center and monetize the hell out of it. Or get a sponsor.

Also, for the short term, Amazon EC2 would be the best option to handle the traffic.

I'll sponsor, or more accurately, DDG will.

the site got twenty two hundred hits within two hours?

In case you haven't watched it yet: this is a quote from The Social Network - http://www.imdb.com/title/tt1285016/quotes?qt1307141

Thousand. Twenty two thousand.

Thousand... twenty-two thousand.

wow

Two hundred twenty thousand. In two hours.

220.000 hits in 2 hours is about 220000/3600/2 = 30 hits per second. The server needs to produce pages in about 33 msec. And all that is average; there could be spikes. Hmmm, interesting. And certainly doable. On my site I use memcached and that produces pages in 3 to 4 msec.

In the movie, the figure quoted was 22.000 hits, so 3 pages per second. Which is reasonably serviceable but still high for a new site.

My guess is this app doesn't need the resources he is throwing at it. If you look at the pages it is dead simple. I can't imagine what was using all the CPU. Would have been good to get a TOP or something to see what was going on. Since it is on Ruby, I recommend getting it on NewRelic and have it tell you where the bad parts are.

Heh, I was wondering what this was when I first saw it a few hours ago. I've seen 5-6 people I follow on Tumblr use this thing already.

I suggest migrating it to Heroku. Push your code, copy over the database, and crank your dynos up. They charge by the hour, so as an example if you have ~10 dynos running (roughly equivalent to 10 thin instances, but slightly better) for 5 days (or however long it takes for the initial traffic surge to subside), you're only going to pay about $50. Then, you can turn down your dyno count and pay a reasonable monthly fee.

There's a bit of magic involved with things like compass/sass and Paperclip on Heroku (you have to use Amazon S3 for storage and there is some other setting I can check on).

Anyway Mark, I have a bunch of apps on Heroku that use all kinds of wacky stuff—including the aforementioned gems—so email me at jarin (at) robotmodehq.com if you decide to try out Heroku and get stuck. I can send over working code snippets that you should be able to just drop in.

I got a few emails about the Compass/SASS on Heroku thing, so I posted it up on my blog in case anyone who didn't email me needs it. Compass and SASS are notoriously painful to get up and running on Heroku, and everyone says to use the Hassle gem but it never works. So here's the SUPER MAGIC WAY: http://jarinheit.posterous.com/the-ultimate-secret-to-gettin...

I haven't forgotten about this. I need sleep (exhausted and have been in bad health lately). Will get back to this soon.

Don't forget to apply for the Thiel Fellowship! Email me for help with your essays (that offer applies to anyone).

Yes, if there is anyone I know who should apply for the Thiel deal, it is Mark. Especially since he has said he doesn't exactly love his college. Thiel should make moving to SF a requirement too.

Mark already applied: http://instagr.am/p/vIqz/?ref=nf

awesome, on Techcrunch 10 min ago:http://techcrunch.com/2010/12/30/three-words/

Hey Mark, first of all congratulations. Do you have a caching layer? If not now is the time to implement one.

Consider adding a revenue stream immediately. Let users pay to see what others have said about friends. Charge 1 facebook credit. Also let users pay to block that access: 10 credits. Also pay to let others ignore the block: 100 credits. Etc...

You'll need to redirect to a facebook page to use credits because they aren't available off facebook.com yet. They don't have many good examples of offsite apps using credits, so they could streamline the approval process I bet. Email me with questions.

I'm not sure all your users are familiar with Facebook credits; I'm not sure you want to set up barriers before you know what you have.

If he isn't making money with an app like this, he has nothing. Ohh and millions of people understand facebook credits. I can tell you with pretty good knowledge that the 1-credit transaction is a good idea for this kind of application.

Certainly this hasn't been proven with apps off of facebook. It's an excellent experiment though.

I thought Facebook credits weren't even available as an option for Connect apps?

If you really want to grow even faster:

1) scrap the standalone website and build it as an iframe app (should be very little work). This will simplify login significantly and will keep people inside FB, which in my experience, makes a world of difference in the viral funnel.

2) replace the "share this link" with a real Stream.publish call. Consider adding like buttons for every single user's page.

3) Allow people to see which of their friends have answers as well.

Then, if you add the 1 credit transaction, you should do pretty well. Beware, however... Facebook has given out a lot of free credits to users and they do not give app devs any payback when free credits are used. These chargebacks can affect actual revenue from small transactions like this significantly.

An offerwall could be a good substitute.

Why did you build it? What was your motivation? I tried it, and I have no idea why would I spam my friends with link so they would describe me. I have absolutely no interest in that, and I would feel really stupid sending anyone link like that.

Did you actually felt the need to have such a website? Or you just figured people will like it? How did you figure that out?

Please, share.

Most people have a strong interest in how others perceive them, including yourself: "I would feel really stupid sending anyone link like that." You're probably part of a culture where publicly asking others what they think of you is frowned on.

It's not the culture. I care about what people think about me, but not as much as asking my friends to describe me it three words.

I think I understand it now. People use this, because they wanna hear people saying good things about them. I think most people would send link to their friends, because they don't feel very confident about themselves.

It's the basic principle of Facebook (or of whatever Facebook turned into) - people post things so they would get positive feedback and they would feel better about themselves.

Well speaking for myself, I am pretty clueless on stuff like whether people think the things I say are generally interesting or generally boring. I wouldn't be ashamed to use Mark's site to try and learn stuff about myself. Of course, I'm sure the factors you mention provide subconscious motivation as well.

Yes, we're part of a mature culture. We do not incessantly need to be digitally liked by others who we don't really know.

Teenagers are always looking for services like this one.

Hey Mark, it's been a while. Nicely done. You said it was just an MVP so I assume you have a product designed beyond this point. So, NOW is the time to do some validation on what those other ideas are. You have a large audience who already know who you are and are willingly giving you input.

Maybe in put down your follow ideas in short little phrases and have people provide three words on that, if they wish, along with three words on their friends. It will give you great insight into what THIS audience thinks is a good or bad next step for you.

This is a site that will not maintain interest as it is beyond a short period unless you add more to it. So the key here is to move forward quickly, even if it is in tiny incremental steps. letting the users help you make some of those choices of what steps to take can only help.

Mark you just started 2011 for me. Very inspiring/smart what you do! Congrats!!!

It looks like it's already been moved to EC2 and switched to Varnish cache and Nginx for static files. I think the decision to tie the "user" view to the main FQDN (http://threewords.me/joe) will come back and bite hard if this project turns into a large user base.

It would have perhaps been better to setup a wildcard DNS/vhost and set it as http://joe.threewords.me. It can still be done but the earlier the better. Time is of essence here.

Regards

Joe

Can you help me understand how this helps with scaling? I prefer this url format myself and noticed it's used rarely by the big guys. If there's a scaling advantage it provides, however, it's not obvious to me.

DNS is much easier to scale out than HTTP. By having full FQDN for users, you have the capability to easily scale out without needing a load balancer in front of say, threewords.me.

So in your DNS zone you can tell it to route all accounts that start with the letter 'a' to server with IP 1.1.1.1, letter b to IP 2.2.2.2, etc. Server with IP 1.1.1.1 will not only run a headless web app but also route SQL queries to a db shard (all relational data for users starting with the letter a).

This level of "user" partitioning can be deeper. So it's not just the first letter of the user but two or three more letters/numbers. So joe.threewords.me would be on a different server than j4mes.threewords.me (That's j.o.threewords.me vs j.4.threewords.me).

Regards

Joe

twitter has servers dedicated to high-volume users without having to use urls/dns like that.

"twitter has servers dedicated to high-volume users without having to use urls/dns like that"

That's why they need so many load balancers w/ failover:

twitter.com has address 128.242.240.148 twitter.com has address 128.242.245.212 twitter.com has address 128.121.243.228

Load balancers aren't flexible to work with especially when dealing with sessions. I imagine Twitter's front end nodes are doing nothing but routing URLs to backends, which are doing the real work. So the front end balancer takes http://www.twitter.com/unixy and routes the actual server-side request internally (where scalability is implemented).

Find your current bottleneck.

Go to static content wherever you can and get it replicated out and hosted somewhere.

consider something like HAProxy (well.. haproxy!) in the stack out front to control concurrency to resources that are bottlenecked - that way you can at least get linear degradation instead of out of control degradation.

EDIT: Drop the dynamic content wherever you can, shut off whatever dynamic parts you can and then scale out horizontally. Then fix the bottlenecks and bring them back online.

About monetizing: you could show the first 2 words only, and charge a small fee for the third ? I guess people would pay to see what is said about them at least.

This is a terrible terrible terrible idea. You don't want your users to feel that you are exploiting them. It will just pi$$ them off. Just go with a bunch of well placed ads.

"If you're curious, the stack is Rails + Ruby Enterprise Edition + Passenger + nginx, which isn't cheap like PHP to host. On the $40 Linode right now but maxing out CPU at 350%."

Interesting. Makes me think that an MVP should be done in PHP or something else that is cheap to host and handles these sorts of conditions well, and then if things look bright build the real site in whatever seems appropriate for the expected demand.

Nah. MVP should be done in the technology you are most proficient at, regardless of how fast/slow it runs. Optimize after you've proven your idea.

I think he just should have hosted on Heroku as he could have scaled it a bit faster.

Seems like a good place to ask this:

are those 'sign in with Facebook' buttons trust-worthy?

I've only clicked one of them, and that was a 'sign-up' button for an event i will be physically attending

everytime i see one, I worry that

a) it's going to broadcast to my entire friendlist that i signed up for this service

b) it's not a REAL facebook button, but something to phish my account

c) this site will know my Facebook login

d) some other party will intercept the login...

tell me it's completely safe?

Haha I recieved links to threewords.me before I even opened HN this morning. I don't know how to solve your fires but I've been asked to tell you to let the text color be customised because otherwise you can make unreadable on certain backgrounds. Maybe you should setup a getsatisfaction-ish feedback/support area?

threewords for threewords.me: hot new cool

Congrats on the App.

Regarding the BizModel. Forget it and grow as long as the hype works.

If you can't: Offer the possibility to add an optional reponse email. An ask for a penny if people want to contact the anonymous poster. Cooperate with FB for FB Credits to see this work.

Expansion: Threewords for all type of content (blog articles, third parties like celebrities, etc)

Just curious: what does it do? (no offense, but it asks for too many things and permissions for me to try it right now)

You don't even need Facebook connect to try it out, just sign up. Send the site to your friends, they give you three words anonymously, you see them.

I need to up the copy on the site.

Then why do you need all those Facebook permissions?

Ah, you're right. This is literally an MVP I threw together in 4 hours.

You can go to facebook and turn off the posting and offline access without disconnecting the app.

Pls Mark, do keep us updated with the Stats

This is exactly why it is nice to have a bunch of credit cards with comparatively high limits, even if you are frugal. There is no investment or other deal which can be negotiated as quickly as just swiping a card and worrying about it 20+ days later. (combined with EC2 or other scale with API services)

Oooh, hey this is bad. If you go to http://threewords.me/username.xml you can see the user's Facebook access token, secret, session key, and hashed password. You should probably remove that view or modify the to_xml in the user model.

It's fixed now.

http://techcrunch.com/2010/12/30/three-words/?utm_source=fee...

;)

FYI: Heroku is fantastic for scaling apps. But if it's your database that's the bottleneck...you've got problems. Because your options with SQL are scaling up or sharding, each quite painful. If you want any advice, ping me.

Perhaps rewriting the database to use MongoDB? Their http://www.mongodb.org/ tutorial took about 10 minutes and is really easy to get to know.

As a DB they say it's designed for SPEED and SCALE. It might be worth looking at. (who's got the time right?) :)

Alternatively, I've gotten a lot of mileage out of using redis as a cache to selectively optimize data piece by piece.

First of all, congrats on your problems :) I've signed up, and things are responsive.

1. Implement caching. 2. Get faster servers to handle your 'debutant' phase' ex: EC2 3. bask in limelight, enjoy!

Congrats on executing the idea and the traffic surge. Looks like you're on the front page of TechCrunch so you'll have more traffic coming your way.

What's your main bottleneck? What's driving that high CPU?

Without knowing more about your setup I'd offload assets to S3 and switch from Passenger to Unicorn.

User-specific assets are on S3... will need to offset the javascript and CSS there too.

Good tip on Unicorn. Any benchmarks on the performance gain over Passenger+REE?

Not sure if there's any benchmarks out there. But I had your similar setup when my app went viral. Switching to Unicorn w/REE replaced an expensive engineyard setup. Learned about it from github (https://github.com/blog/517-unicorn)

But something you could do immediately to help with the load is to run haproxy on your current linode and funnel some of the traffic to a new linode that you spin up.

Mark,

How did it go? Is the traffic still sustaining? Also tell us if you have been able to monetize it? Share your experiences with a blog post.

Cache the hell out of it! I'd spend all of my time looking into caching mechanisms for Rails right now.

"502 Bad Gateway" - hosting troubles.. Can you explain a bit more about what threewords.me is?

It's a site where you can ask people to describe you, in three words.

I am not sure how long it is going to last. i mean you will run out of friends to write about, Isn't it?

then it starts suggesting people. apply it to more than people. then it starts suggesting more than people. you get the picture.

Yoggrt.com seems like a decent ad network if Fusion Ads and Deck Network won't bite.

InfluAds.com also does

Clickable: http://threewords.me

turn on mysql connection pooling and install Google Web Accelerator, you'll have to switch to apache to do Google Web Accelerator though.

could not you make it pull my photo from facebook as well? also ad an option of inviting friends from facebook?

yo mark, congrats. So what are your next steps? Grab an advertising network quick and throw it on there for now.

What does MVP stand for?

Minimum Viable Product http://en.wikipedia.org/wiki/Minimum_viable_product

short term: heroku

long term: app engine (just port it to webapp framework)

Why?

Heroku in the short term b/c you wrote it in rails and you can easily scale it up or down based on virality.

App engine after that b/c it should be fairly easy to port and it's an extremely headache-free way to host an app. Now that they have always on instances, it's very fast and scales efficiently and effortlessly.

Great idea! BRB, working on sixwords.me

Heroku.

You are on Techcrunch!!!!

I will pray for you.

how is the load now?

That's why I use google appengine. It would have handled what you describe without even going over the daily free limit. Of course, you wouldn't be cool as rails.

You could be cool as Clojure, though: https://github.com/gcv/appengine-magic

Whats the start-up time? i.e. when google spins up a new instance to handle traffic, how quickly can it come up?

In the free configuration, I've clocked it between 4 and 6 seconds. (Sometimes up to 10 seconds, but not recently.) With the 1.4.0 release of the App Engine SDK, the cold start time has become less important, though. First, you can pay $9/month to have three instances running all the time. Second, App Engine can send a "warmup" request to your application.

minimize the server work; get some cheap nodes at rackspace cloud; use one node for sessions in memcache; use two nodes for mongodb (master/slave) read from both; try to minimize xss/html/js and gzip everything; cache as much as possible; maybe even queue inserts/updates with redis and a cron;

thats how i survived a mini fb app thatwent from zero to 44 million users in 7 days... rackspace cloud + memcache + redis insert&update queue + round robin. after a month i even killed the rackspace cloud interface and api with over 80 nodes and growing. had to get the memory limit on my account raised twice. peak were around 40k req/sec. after the second month we moved it to four bulky servers (16 core, raid5, 32gb ram) each cost around 12k dollar.

the whole secret is caching. everything, everywhere and as much as possible.

which was mini fb-app, if you don't mind me asking?

Seeing the title on the front page, I came here to post "Monetize it, baby!". Now that I've read more detail I'm in the camp, "Put out those fires NAOW and put some extra water buckets nearby too."

Then monetize it, baby.

First, do not put ads on your site as it will annoy the users. Rather, take feedback and see what it takes to get more traffic. I guess you can double up your existing infrastructure to handle any surge in traffic. You do not want your users to see your page down due to excessive traffic.

So... what is it?

I just don't get it, what problem does this app solves anyway? Plus, HN has become a promotional playground, this really piss me off.

what do you do? stick ads all over the place! what do you think, you have a business there? like twitter? yeah, sure, raise another 200 million...

make fun of twitter, and you get down-voted. I know, I know, Google will buy them for 4 billion in 2011, but that is the only exit strategy they have. No biz model there, sorry.

Just to let you know why I've downvoted you twice: Twitter has millions in revenue a year.

Thanks for cheating (down-voting me twice).

10 million, or something like that.

From ONE customer. who could well drop them.

So, they're breaking even (for now).

And all this while they have raised 360 million in VC money (crunchbase).

And nobody has a clue how they will make REAL money.

Or do you?

It's not cheating. You made two comments that added nothing to the discussion. They're offtopic and just full of vitriol.

They have more than one customer, first of all. They have two. ;)

But seriously, if you can't imagine how a company that's collecting real-time data about the thoughts of millions of people around the globe might make money, you might want to pay more attention to Google...

Cheers.

so, now it's three down-votes (but it's perfectly ok on three comments :)

They could perhaps be useful (I personally do not find much value in the "real time web") - to either Google or Bing.

So, one of the following is going to happen:

-Google buys them

-Bing buys them

-Google and/or Bing continues to keep them afloat

-none of the above, and the service gets either shut down, or bought up for a fraction of its current valuation.

Nope, once I reply to you, I can't downvote you.

p.s. up-voted: you're right on the off-topic and vitriol part.

Twitter's been profitable since their Google and Bing search deals lit up around last November.

yeah, "profitable". Let's say at break even.

And yet somehow they needed 200 more million, for a total of 360 million sunk into that ship. Do you have a clue of how profitable a company in which 360 million dollars were invested needs to be? Do you really see them ever getting even close to being that profitable? How?

Having worked from Twitter's office in late 2009, I'm privy to a bit of their financials from that time and onward. They're definitely profitable since then.

Profitable companies often raise money... I don't see why this is out of the ordinary. It lets you scale and make more money more quickly. Another startup I worked for did the exact same thing as soon as they became profitable.

yeah, sure, for example, they could sell data about their tweets to search engines other than Google and Bing. Oh, wait, too bad there are none - or not in the parts of the world where people use Twitter, anyway.

so, yes, 200 million on top of 160 million already invested for a company that is making 10 or 20 million at most and that has no other way of making money in sight is, well, yes, I would dare say quite out of the ordinary.

all right, guys, group-think wins, I lose. Have a nice day!

Where are you getting this 10-20mm number? That wasn't even accurate (it's too low) a year ago.

The value of the deal with Google alone was 30mm. That's just a single facet of their monetisation strategies. They've also introduced Sponsored Tweets and a bunch of other things since then.

Fun fact: Twitter actually violated their SLA with Google and Bing before the first month was even up. They guaranteed a maximum amount of time between a tweet being published and it being available via the commercial API access given to Google/Bing.

FYI, I am not downvoting you. I can't even downvote yet. I don't even like Twitter and think it's mostly a waste of time.

I don't know, someone made a comment in reply to me the other day that said "Twitter could be another Digg" and everyone seemed to love it.

I sure do :)

I somewhat like Twitter. I avoided it for a long time but I've come to enjoy and accept it. Facebook, on the other hand, is one of the worst entities in the entire world.

Have you seen this? - http://business.twitter.com/

how much money do you think they can make with that?

I don't really want to offer my random guess against your random guess of "not much money". Why not wait and see? What vested interest do you have in seeing Twitter fail?

I hope Twitter makes a lot of money and lasts for a long long time. It will be good for the world.

I have no vetted interest in seeing Twitter fail, nor do I think it will. I think Google and Bing will keep it afloat for a long time until they figure out how - or if - they are really interested in the so-called "real-time web".

A subdomain isn't a business model.

I created at stupid app a week ago, called http://wonklyrics.com where the point is to enter a songs lyrics, and change one word to make it more... funnier.

Sadly to say, it's received a total of 300 visits. :-P

Just a heads up, the app doesn't protect against xss at all. For example: http://wonklyrics.com/24

Reply on news.ycombinator.com