Scaling HipChat Using ElasticSearch And Redis | Hacker News Reader

Scaling HipChat Using ElasticSearch And Redis

highscalability.com

53 pointspscsbs16 comments

    Stats
     * 60 messages per second
    ...
      * 26 front end proxy serves. Double that in backend app servers.

78 servers for 60 messages per second?

We have one MSSQL box at work here that is regularly doing 2,500 queries/sec and that's an OLTP workload so easily more than 60 writes/sec.

serves makes me think that someone didn't proof-read the article...

How did it even make sense in your head to compare an OLTP workload to a chat service?

The proxy servers are there to terminate the large number of persistent connections. Of course it's possible to do it using less servers, but given the Hipchat guys are smart (disclosure - I'm an Atlassian and know the internals) I would give them the benefit of the doubt rather than engaging in armchair architecture.

The point was to ask whether the 60 in 60 messages per second is wrong by comparison.

The frontend needs to keep open thousands of persistent connections; 99.9% of which will time out with receiving a single message.

Practically speaking the number of frontend servers is related to the number of active listening clients and not messages.

Only 60 messages per second does seem very small from a business perspective however.

If you take the frontend servers out of the equation it's still 52 servers for 60 messages per second.

My guess: the 60 messages per second is an average. Monday morning in the US timezones might see several multiples of that. Weekends could be a lot lower.

It's definitely very bursty. On weekends and holidays things are much more quiet. During peak load we'll be in the hundreds/sec. Also keep in mind that chat messages don't actually make up the majority of the traffic we serve; it's presence information (away, idle, available), people connecting/disconnecting, etc. (I'm one of the HipChat co-founders.)

And? The article makes no mention of what each server is responsible for, how much redundancy is built in (this is AWS where nodes die on a whim), whether all of these servers are used for production or if there is a spread across dev and staging environments, etc.

When all of these factors are considered, 52 servers isn't a huge number.

Looks about right from their graph:

https://s3.amazonaws.com/uploads.hipchat.com/10804/368466/10...

That traffic is going to be highly bursty, though.

May I ask what kind of company you work at where you have that kind of queries/s?

Australian childcare software provider.

I imagine an outsider wouldn't equate "childcare software" with "database intensive" but the online reporting childcare services have to do is quite involved [1].

Their online reporting to the government is a separate concern to day-to-day tasks in their childcare service. So childcare software has to do more than just interface with the online reporting interface.

[1] http://docs.education.gov.au/system/files/doc/other/service_... (The attendance information of children is the most complicated part, see page 160 for what an attendance record looks like.)

Is it hubworks?

No. I don't know what RDBMS HubWorks use but to my knowledge they run their software with AWS and I think there's better fits to AWS than MSSQL.

On my shit dev box, localhost Elasticsearch is indexing (AKA writing) @ approx 2,500 msg/sec.

Can you still process that many while holding like 2M connections open?

> .5 terabytes of search data.

500 gibabytes isn't cool enough ?

Reply on news.ycombinator.com