How NYC startup Tapad uses Scala and NoSQL to process 25k requests/sec (for now) | Hacker News Reader

How NYC startup Tapad uses Scala and NoSQL to process 25k requests/sec (for now)

g33ktalk.com

51 pointspetesoder19 comments

The fact that they use Spring to wire their application together is telling that Scala in its current state isn't meeting everyone's wants or needs at the architectural level. If you follow the Scala mailing lists you'll find a lot of anti-Spring sentiment, and I totally agree with them. I've hated Spring since I first saw it years ago. Why use a statically typed language if you're going to throw away type safety for the important task of wiring your application together? Why use a modern and powerful language if you're going to, again for important parts of your application, rely on a horribly slow and gimped XML DSL?

One of the reasons I think people still use Spring is that XML configuration files don't require recompilation. I think that to address the Scala developers or community need to make it incredibly easy for you to run Scala scripts as part of your application. This will then allow people to use Scala like a dynamic language and allow them to deploy Scala configuration without needing to recompile.

Also, I think work needs to be done in the community to document accepted design patterns one can employ to wire together your application in Scala in a way that solves all of the use cases Spring configuration is currently used for. This, combined with making it easier to run dynamic Scala, and people will never feel the need to turn to Spring again and the world will be a better place.

They probably started with a java backend, I think spring is a total mismatch also for scala there is some movement in the community, personally I think we should move away from IOC frameworks they are really better for languages that are so inflexible like java

You make some very good points here. Although we started from scratch with Scala, we decided to use Spring for several reasons. I mention one in the talk - we initially adopted Scala for the language and syntax itself, and it made sense at the time to mitigate some technical risks by sticking to a well-known and familiar architecture / wiring / configuration model.

As we have become more familiar with the language, FP and the Scala idioms, we have definitely become less comfortable with our Spring "legacy". Initially, we felt that @BeanProperty annotations and setter injection saved us some boiler plate, but now most will shudder with having uninitialized vars. Constructor injection is somewhat better, but the XML DSL of Spring is still ridiculously verbose. Luckily, this doesn't bother us much in the day-to-day work, we spend most of our writing idiomatic Scala.

As our platform and the entire Scala ecosystem and libraries have evolved, the Spring and JEE "legacy" has become much less prevalent in our system than it was originally. We use a pimped version of Spring JDBC, Spring contexts for wiring, a Servlet container, but that's about it. In fact, we could probably easily rewrite the whole Spring config in Scala and drop most of the Spring deps.

I couldn't agree more with you in that the us in the Scala community really should come up with an accepted way of handling configuration and wiring. Having worked with more than a few Scala libraries, I can definitely attest that the plethora of configuration styles used can be a pain to manage (and dependencies -> configgy-nologgy, anyone?).

We regularly talk about this topic in the NYC Scala community, but we haven't come up anything good yet. Personally, I strongly believe that dependency injection is the way to go and that singletons and factories (which are fairly common in Scala libs) are evil. Most long-time Java devs will agree that we've been down that road before (gotta love JNDI)... That being said, we did not adopt Scala to fight for status quo, so we'll keep an open mind.

Dag

Have you looked at graph databases and how would you compare them to your with Citrusleaf document store?

A similar-sized ad network, CPX Interactive is running ad requests on InfiniGraph: http://www.youtube.com/watch?v=88HCKQ73nFU&feature=relat...

Seems like graphing engine would give you a lot of native search / lookup functionality for free.

We don't need to do any advanced querying or search during the real-time bidding request process - it's just raw lookup based on "cookie ids" (we're not always using cookies). Simplicity of operations, horizontal scaleability and consistent, predictable performance are the key characteristics required.

Other parts of the system have very different requirements and yes, we are considering graph oriented databases for our cross-device analytics product.

Great stuff there, but you didn't ask about the most interesting stuff: how they determine what to bid on an ad! Of course, I'm biased as a machine learning guy :)

Having the transcript is a really nice touch. It's much faster to read than listen. I hope you can keep that up, though I realise transcription takes a lot of time.

Thanks for the feedback - much appreciated. Yep, figured the transcript was key since video requires commitment, and we definitely aim to keep that up.

Curious to hear if people benefit from the mp3s, too, and to see if we get many click-throughs there.

Maybe you should come talk to us? :)

I've exchanged a few emails with Toby about this topic. He knows what we do, but he's very cagey about revealing details of what you do. :)

Great content, and I love the length — I get overwhelmed when I see a half hour or hour long tech video. I literally have to schedule time for those... 7 minutes is the perfect depth into most topics.

Scala scene is heating up in NYC, Tumblr seems to be the biggest in the area, some of the other startups like FourSquare and ideeli are using it also. Word is some of bloomberg.com is on it also

Scala strikes a good balance between flexibility of Ruby, power of functional languages, and type safety of staticly typed languages like Java. I've seen as much scala adoption from rubyists as I have from java devs.

Tumblr is using scala? Any idea for what? I know ideeli was using jruby. Did they start using scala as well?

Tumblr is doing their new backend services in scala, they are starting to ramp up hiring, they want to do a lot of stuff like twitter so your seeing them do a lot of hbase also. Ideeli told me they were considering using it for backend services. Obviously I was interviewing lately so I got to hear all the recent gossip ;)

insert boilerplate request for skimmable, searchable transcript or slides to accompany video

Sure - in case it's more convenient, you can get a transcript of the entire interview here: http://g33ktalk.com/post/10782097232/dag-tapad-interview-tra...

Awesome, thank you!

Reading it again, my comment while intended to be flippant just came out as churlish - apologies for that, and thanks for sharing this information!

Good content, well edited.

7 minutes is the perfect length. Even though I have no idea what the fuck these nerds are talking about.

Reply on news.ycombinator.com