How ZenPayroll processes billions of dollars in annual payroll

I was also surprised. You hear finance and the last thing you imagine is Rails but there ya go!

I also work on a payroll system (Rails/postgresql) based that's used for internal company use with hundreds of employees and multiple states. The hardest part is the taxes and benefits that's handled by external vendors.

Doing SOA in Ruby isn't so bad. Nice thing about services is that you can keep the the total lines of code low in each service. Ruby is better when you don't have giant apps.

JRuby is also an amazing piece of technology. There are plenty of instances where you can change the engine and get all the performance enhancements of the JVM whilst the flexibility of Ruby.

Note: I have no idea if ZenPayroll uses or has considered JRuby. This comment is specifically about improving the performance of Ruby.

Disclosure: I am an engineer here with ZenPayroll.

In terms of language choice, it is actually quite nice to see definitions of taxes, rules, and forms for different states and counties in a dynamic language like Ruby without verbosity of other languages like Java. As mentioned in another block post[1], when launching to new states, we paused and built tools to help us automate the verbose tasks, and Ruby really helps us there. We have good test coverage so we have high level of confidence when refactoring (which we do frequently).

In terms of JRuby, we are only using it on one of the components that needs to talk to a Oracle DB using JDBC.

We are early in the service oriented architecture path and will be learning (hard) lessons a long the way with mistakes and from new talented engineers joining us. What I am most impressed with this team of engineers is a collaborative culture and willingness to go extra miles for customers and for ourselves.

That said, I wouldn't be surprised that one day ZenPayroll technical stack would be polyglot.

[1] http://engineering.zenpayroll.com/benefits-of-writing-a-dsl/

(I'm not a ZP employee, but I have worked in environments of significant scale a fair amount. The below is just my opinion...)

#1: The usual approach is something like remove app servers from the load balancers' rotation, upgrade them, run some sort of tests to ensure the upgrade is ok, add them back into the LB rotation. Some environments might use user traffic as the test step (e.g. send 5% of your users at the newly upgraded machines, watch monitoring to see how response times and error counts behave).

#2: The answer here really depends on what scale you're operating at. DB replication (e.g. read slaves if your traffic is more read-heavy that write heavy) and sufficiently high-spec machines combined with intelligent use of caching can take you a fairly long way. I hesitate to give hard numbers, but off the cuff probably 99% of sites will never need more than this. There's not really a recipe to follow for that, the data needs and design/ops parameters of an organization tend to be fairly specific to that org; more or less it's good schema design, thinking hard about how fresh data has to be at the edge, caching what you can with careful management of expiry times, and having an ops team comfortable with managing replication. Beyond that scale you start getting into globe-spanning multiple-datacenter voodoo rocket science. :) Ask amazon, google or facebook (some of their work has been mentioned publicly, things like cassandra from fb and various papers published by google/amazon).

Perhaps. But orders of magnitude more payroll has been conducted by supposedly prehistoric banks for the past century or 2.