How ACH works: A developer perspective | Hacker News Reader

How ACH works: A developer perspective

engineering.zenpayroll.com

179 pointsedawerd94 comments

This made me cringe:

> Your [bank] will set you up with a secure FTP server onto which you’ll upload ACH files.

Automated financial transactions, with lots of money involved. Secure. FTP server.

Of course I know that FTP is still considered "secure" for some business applications but this is absurd. With all the security (mis)features the banks have put forward with card payments in order to shift liability to account holders, the very core of inter-bank transactions then runs on something as insecure as FTP.

The shoemaker's children have no feet.

> This made me cringe

As well it should. But it gets worse. Much, much worse:

http://blog.rongarret.info/2010/11/personal-banking-nightmar...

It often amazes me that the U.S. financial system works at all.

Maybe. Here's a slightly different perspective. First take the case of ftp'ing/sftp'ing files to a bank.

There are multiple layers of security in sftp - not just the username and password. Source IP is checkec in a stateful way so spoofing is hard even if one somehow manages to get the credentials. Furthermore the files will not be processed unless there is an out-of-band message (typically an email, could be a phone call too ) with certain info also required in very specific format which must match the contents of the file. So there you go. 3 layers of security with one layer being completely out-of-band with the other two is reasonable enough for the risk involved. (There are other measures hard coded e.g. a dollar limit which basically "cap" any fraud but I won't go into those)

Now for the missing (or atleast MIA) transaction you allude to on your blog: Just because you were not told how the money ended up where it did doesn't mean people didn't know. Bankers are steeped in "social engineering" and trained from day one to only give out information required by law and really nothing else. Whether you use Fedwire or Swift (or really any other RTGS = real-time gross settlement network ) there is always a trail. Always. In your case it was probably Fedwire - and the bank I'm guessing was BofA or maybe Chase. Regardless if the sender's account is an individual account (not business) they are still protected by Reg E (as well as FDIC insurance). Reg E as you may know basically puts the onus of returning the funds to the sender's account on the bank. The problem is the timelines or the lack of it. Since banks are generally allowed upto 30 days (I don't know if 30 days is hardcoded ) in law they take their sweet time in tracing. And when they find out - they almost always do - they deliberately will not share details with you to protect themselves from liability.

The bottom line is ACH/FedWire moves trillions of dollars - hundreds of billions a day - and barring a very small percentage they all work.

> Just because you were not told how the money ended up where it did doesn't mean people didn't know.

That's true, but there was a lot of additional evidence that they really didn't know:

1. They said they didn't know.

2. Two weeks passed during which they could not find the money.

3. The money was only found after the person in whose account it ended up contact me (not the bank!)

And, BTW, I actually found out later how the mistake had been made: the original wire form had been filled out wrong. It was supposed to be a for-benefit-of wire, but the form was filled out as if for an intermediate-institution wire. And the bank employee who entered the data was apparently completely clueless because they entered an account number as an ABA number (or maybe it was the other way around, it was a long time ago). But it was without a doubt a total clusterfuck from beginning to end.

> barring a very small percentage they all work

Yes. As I said, this never ceases to amaze me.

It works primarily not because of technology but because of inter company agreements, contracts and regulations.

That's a fascinating post, especially the comment from Don Geddis:

> ...I eventually reached the conclusion that "check clearing" doesn't mean anything at all. What there really is, is a web of trust among large financial institutions. They take actions which work 99% of the time, and then they have correction procedures for the 1% that fail...

> ...their primary concern is that I'm not a con man who is going to take money out of the system. As long as the cash is EITHER at their bank, OR at the original bank, they can afford to not be completely sure just where it is, and to work that accounting out over time.

> This system is totally unlike how computer scientists would design it. They would want guarantees that the money really was only in one place, and there would be a thing which is a "transaction", where the money reliably went from one place to another. A wire transfer is something like that, but it's really bootstrapped on old-time banking, which is a trust relationship with repair procedures, not a guarantee of freedom from error.

I don't know anything about ACH, but the author could be referring to FTP-over-TLS ("FTPS") or SFTP (unrelated protocol over SSH), both of which use the initials FTP but aren't as wholly insecure as vanilla FTP.

I've seen https and scp as well. I'm not sure I've seen plain vanilla ftp in years, and certainly not without pgp encrypting the files first.

Our bank uses SFTP, which is really just SSH (nothing to do with FTP).

yerp, fixed width nacha files exchanged periodically over sftp. Built for/by cobol based banking systems back in the day and continues to live on.

Its actually quite secure, just a folder where a job picks up a file for processing... It doesn't have access to the ACH or banking network. The transaction still needs approval which will come from an another method, usually with strong encryption.

The author probably had it this way because they did want to spend the money for a 3rd party integrator which would process the transaction via a web service.

  Hey I can't believe EDI is still alive but it is....

...not to count all of EDI's bastard children that are 'easier' [generally by taking out some/most of EDI's redeeming qualities]. shudders I had to implement one of those in 2014. :|

So the third party integrator which processed the transaction as a web service... would then turn around and drop files on an FTP server at a bank, to _actually_ make the ACH happen?

I think maybe OP is essentially a third party integrator?

No... 3rd parties like cybersource act just like banks from the perspective of the network and can initiate transactions. They pay big money to the financial institutions for this level of access... But from a dev perspective they provide a defacto standard for transactions across many institutions and networks

I happen to work with ACH/NACHA files and send them to/from various locations. The file spec outlines that there are "control" records with various values that are supposed to be used to validate the batches in the file and the file itself to prove the file is internally consistent. I have often found that when receiving the file, these control records are hard-coded and invalid. And when sending files, some vendors don't even bother to look at the control records. It's a whole different world.

Depends on who you are working with, I worked with a Bank once that gave me crappy fails and it failed because I had a validator. The new bank I work with follows the specs strictly. So if you are getting invalid files, demand that the originator of that file fix it.

Let me just observe that "I am not sure I need to have access to the $GUESS-test-results directory." "Yeah, of course, your stuff is in the directory next to it." "I am aware of that, but just wanted to observe, for HIPAA compliance and basic privacy reasons, you probably want to restrict access to it a bit." "Oh yeah, but that vendor's software is really wonky, so we just 777ed it to get it working." has actually happened.

I was pleasantly surprised. This conversation implied that 777ing it was a considered choice, as opposed to having just happened because YOLO. This puts them much higher up the healthcare IT totem pole than you would hope.

No cringing.

Secure-ftp is not the only security measure used in transmitting ACH files. There other measures too - including but not limited to: Checking source IP in a stateful way, Out of band confirmation by email/fax/phone, hardcoded limits and out-of-pattern detection mechanisms. I won't mentioned the specifics of some of these since this is a public forum but defrauding the ACH system by hacking into sftp is neither trivial nor scalable.

Again no system is full proof but ACH fraud is typically 1/3rd of the fraud in Credit Card networks and has been the same percentage for decades.

Please, stop cringing. I work in this field. It's secure, I upload file to the banks SFTP, yes, Secure FTP. And the NACHA file is PGP encrypted with the bank's public key. Firewalls are in place, the bank only opens up to a specific host within our network and denies everyone access. We open up a port for only the transfering host to the banks SFTP. Their SFTP is not a regular SFTP that you can just browse. They have a folder with -wx permission. You can cd into it, place file but you can't read it. Stop cringing. Banks are not that incompetent.

I used to work for an ACH organization in security, as the guy who set up the credentials, submitted the firewall changes, configured the FTP server, created the PGP keys, troubleshoot connectivity, you name it, all this. I know too much about the subject and I would say "its been 5 years I'm sure they have made it better" but after spending those years there, I know that statement is false.

To say it's secure is a stretch, so say it's completely unsecure is also inaccurate. There were multiple improvements I recommended to the service, but I think I only got 1 or 2 approved. Lets just say this, the front door is secure, but the once you're inside, its not so great. It's not incompetency, its just we (the security guys) are always fighting an uphill battle against change.

Also, I could tell you worst stories about other services all the banks use that would make anyone cringe worst than this, some simple cross bank privilege escalation, oh yeah and the developers said thats not really going to happen... I resigned within a month of that.

I have no doubt this is true of many organisations.

As the client, though - we uploaded via SFTP, the connections were IP restricted and the files were PGP encrypted.

I know that doesn't address what happens after we send the bank the file - but that's not our concern, right?

Access to the FTP servers are IP restricted and everything is encrypted in transit and at rest on the server via PGP. In my organization the transfers where via FTPS not SFTP, big distinction, the FTPS implementations can be not as secure by default as SFTP. But yes, once it's on the ACH processors servers it's their responsibility and not your compliance issue. They will pass an audit, but from a security point of view, they could do it better in a few areas.

I used to work in this field as well... it's laugh-out-loud insecure.

(throwaway account) We had to get/put data to a bank. Our software architect suggested FTP. He even knew we had fancy XML gateways to enforce security and validation. WHY?!?

Other story: I've add access to a FTP server which also served as a way to submit JCL at an escalated privilege to an IBM server!

Mostly "enterprise" security is a joke; it depends on the people not the technology.

Thank you. If it's SFTP, then it's a different beast and that changaes a lot.

Yes, I know the difference between FTP and SFTP. In this case the technical details are important: SFTP is effectively a bolt-on file transfer protocol which requires an already established (authenticated) connection. It is most easily used with SSH, but as far as I recall, it could be implemented to work from any protocol that has the concept of a session. (And if my memory serves me right, SILC implemented it as a logical replacement for DCC.)

The other security measures you list also make me feel better. From other posters I have already learned that NACHA transfers have an integrity check file which may be, in some systems, ignored. If the files are indeed PGP encrypted, then that may be less of an issue. The message integrity checks in PGP are certainly robust. :) [Corruption-in-transit becomes a moot point, and the same applies for route hijacking.]

I give you wholehearted thanks, and want to offer an apology for my earlier tone. However, I still have a reason to cringe.

Just let me cringe at the article author instead.

I realize the article didn't say SFTP, but that couldn't be gathered from "secure FTP," as that's precisely what SFTP means?

SFTP is SSH File Transfer Protocol, and is actually its own protocol (not FTP over SSH).

There's also FTPS, which is FTP with TLS. That's closer to "secure FTP," but as @segmondy pointed out, this isn't what they use.

When I read it I thought the author was claiming plain FTP was secure too.

All of the banks I deal with use FTPS (which is called "secure" ftp). I have yet to deal with one that uses SFTP (ssh-ftp).

http://www.differencebetween.net/technology/internet/differe...

I understand the difference, yet the term "Secure FTP" is most often used to refer to SFTP and not FTPS, in my experience.

From wikipedia (http://en.wikipedia.org/wiki/File_Transfer_Protocol#Secure_F...) "The SSH file transfer protocol or secure FTP (SFTP)..."

Multiple google searches also yielded similar language.

As the data transfer is usually implemented, sftp is okay; and actually the files might as well be sent over common email, if they're properly formed (encrypted;signed;format that doesn't allow replay attacks) - breaking that ftp server could only delay service, as you'd know that the transactions didn't go through as intended and would use a backup channel to send them.

Virtually all banks run on batched FTP processes. This isn't unique to ACH at all.

Not FTP - Secure FTP - SFTP.

When I was involved with this it was SFTP for the transfer, to a write-only folder (you couldn't list contents) and the file uploaded had to be PGP encrypted with a key the bank gave us and we verified over the phone before starting the process.

Even if it had been email, or plain FTP, the file was encrypted and only the bank had the key... even we couldn't decrypt it, due to the nature of Public Key encryption.

Plain old FTP is not considered secure by any means, and is largely disappearing - SFTP is standard now.

Likely it's not SFTP as in the secure file transfer protocol related to SSH, but is actually http://en.wikipedia.org/wiki/FTPS

It's sftp in my experience. And all keys have two year expiration, which is stressful because ssh keys don't have a real expiration do they just send you an email saying "give us new keys" and you have to hope the cutover goes smoothly.

Slightly OT, but OpenSSH now supports the use of signed certificates, giving you the ability to expire and re-sign credentials. The feature was added recently, so I'm confident they're not using it yet.

> Plain old FTP is not considered secure by any means, and is largely disappearing - SFTP is standard now.

Thank $deity for that.

> The shoemaker's children have no feet.

What does this mean?

It was actually a direct translation from a play's title. One I never managed to find time to see, to be honest, but wanted to.

Slightly more easily recognizable version might be "the shoemaker's children have no shoes".

If you want to get into the nitty gritty of the format of ACH/NACHA files you can buy the spec here:

https://www.nacha.org/nacha-estore-rules

But you can find parts of the book online at various places. Many financial institutions support NACHA files, but you'll also find many that violate the spec. I happen to have implemented a C# NACHA implementation recently and it wasn't a lot of fun. And there is a huge difference between ACH receipt and ACH origination.

If you don't want to wait for part 2, NPR had a decent podcast about how the current American ACH system works, and why it hasn't moved out of the 1960s:

http://www.npr.org/blogs/money/2013/10/04/229224964/episode-...

good podcast. its apparent that the ACH network is considering intra-day settlements and with Check 21 as an option, money can be moved/settled quicker

The main reason it hasn't moved out of the 1960s is it works pretty damn well. It's a bit slow, but money goes where it needs to go, and there's a good audit trail to trace in cases of fraud.

It's crazy slow. Most ACH moves through a modern stack and then into COBOL that runs nightly batch jobs that then update the stuff in the modern stack and in on its way. Nevertheless, it takes the nightly batch job.

Honestly though, they processed 22 billion payments totaling 38.7 trillion dollars in 2013.[0] When you're dealing with numbers like that, there's an argument to made in favor of battle-hardened COBOL batch processing over some real-time rewrite.

[0] http://www.prweb.com/releases/2013/ACH_Volume/prweb11740323....

One of my banks, every night between the hours of 12 am and 6 am America/Chicago (local), notes that they are processing the day's transactions or something, and my balance might not be correct. I presume they're handling ACH transfers during that time.

That said, during that time, my balance has read any of the previous day's balance, the next day's balance after processing all pending transactions, $0.00 (this one is particularly worrisome), and amounts that cannot be obtained by adding any combination or subset of the day's pending transactions. And that's not mentioning that the uptime for this is effectively 75%, assuming nothing is going wrong.

It could be faster. I could enjoy not having to worry that I might lose vast quantities of money because somebody got the number that I both must keep secret and give to everyone I do business with. I could wish for bank websites that understood how SSL works.

Yes, for what it does, perhaps it "works pretty damn well". But the requirements have changed since 1960.

Just curious. How hard is it for a bank to agree to be your “Originating Depository Financial Institution"? We currently use Balanced for ACH payouts, but it sounds like we could actually do this ourselves and significantly speed up the process as well as cut down on fees.

It varies depending on your risk profile and how much the bank wants to do your transactions (dollar volume of deposits matter, and fees). If you're coming in as a customer, it may just be a matter of being a commercial customer and paying enough fees. If you're trying to come in as a third party processor, it's more involved. Relationships help there.

You might need significant volume (hundreds or more per day) to make it worth switching from Balanced to something else for outgoing payments. With enough volume (10k+/month), you might get under .10/transaction. Balanced doesn't seem like they're particularly slow about outgoing payments. (from a quick scan of their site) The slowness comes from the ACH network where nothing is ever confirmed settled, it just hasn't come back returned yet.

My main issue with Balanced is that since I don't use them to collect payments, it takes forever to get the money moved to them. And it is a manual process.

So every Monday morning I get a file with payments. I have to move that amount over to Balanced. This can take 3 to 5 business days, but almost always takes 5 or even 6. Meaning I cannot pay my customers until the Friday (5 days) or the Monday (8 days) later.

Also, Balanced cannot process ACH for payments from clients. Having to verify an account with micro deposits is simply not workable for our clientele, many of which do not want to take credit cards because of the high fees.

While our volume right now is too low, it is interesting info to have, and something I'll look into further.

Thanks!

Basically, anything that's not next business day is a delay due to risk. (well, except for online bill pay, which can be printed checks that are mailed out.)

Depending on the risk profile of the business, we've done an array of things from 0 day simultaneous ACH credits and debits to 3 or more day holds on incoming funds (via ACH) before credits can go out. Obviously, the riskier the business, the more expensive it is to go fast. 3 day holds are pretty common, since any NSF sort of return is supposed to happen in that time frame.

Hi theflyingkiwi42. I'm a co-founder of Balanced. Happy to answer whatever questions you have about our product and hear how we can make Balanced's ACH products better, but don't want to derail this thread. Would you mind commenting on one, or more, of our ACH related GitHub issues [1]? Or you can email support@balancedpayments.com to expand on your thoughts here. Cheers!

-jkw

[1] https://github.com/balanced/balanced-api/search?q=ach&ref=cm...

Building a relationship with an ODFI generally takes many months (3-6) and multiple audits (sometimes including PCI), in addition to specific risk and capital requirements. Marketplaces generally have higher compliance & capital requirements, as consumer debits are one of the riskiest ACH transactions. For their customers, Balanced (and others) remove a lot of the compliance & business overhead in exchange for a higher fee (most ODFI's charge less than $0.15/transaction even at low volume).

Are pinned debit card transactions considered ACH transactions?

None of the payment processing providers differentiate between the two.

So, an entity that's set up with an ODFI can initiate arbitrary debits to arbitrary accounts, with no action by the owner of the debited account. What happens if that privilege is abused? Is it like chargebacks with credit cards? What if the debit is so small the victim doesn't notice?

There is a strict system in place for violating the rules:

https://www.nacha.org/NSfines

But if things occur on accident, there is a "Returns" process for handling it. It looks like the author intends to cover that, at least partially, in the next installment. Here is a list of return codes if you want a hint:

http://www.mainstreetmerchantsolutions.com/ach-return-codes

There are all sorts of policy-based resolution methods in the ACH world. But there is nothing technologically restricting arbitrary debits/credits to arbitrary accounts in arbitrary amounts.

One policy meant to deal with this is called "Velocity limits" which is basically a rate-limit of how much the OFDI will underwrite your account for. i.e. You can debit up to $50,000/day and then your OFDI will cut off further ACH transactions until some of your outstanding ones have settled.

Another policy is that (for Check21 checks processed via ACH anyway) the checking account owner has up to 60 days from the statement date to dispute the charge, meaning the ACH transaction can be reversed up to 90 days after it was initiated.

It's straight out of the 60's. And on one hand, it's cool that everybody uses this system and nobody notices how old it is. On the other hand, it's freaky when you realize that every time you swipe a debit card or write a check, you've basically handed over all the credentials needed for that person to debit your account as much as they like (until they're discovered).

Also, remember that automatic deposit implies automatic withdrawal.

ACH systems are by far the craziest API integrations I've ever had to do.

I'm an ACH product manager who's worked the user side of ACH for years, and I've never seen a conversation like this. It's amazing. Even though most of it went off on a file-transfer tangent, I've picked up some really interesting tips. Like, I never imagined there would be stuff about ACH on GitHub. BTW, ZenPayroll is an amazing product and I want to work there.

Great intro to a near-ubiquitous payment format (in the US - outside the US, it looks like ISO 20022 [1] is taking over).

I'm looking forward to seeing details about payment confirmation/rejection, and possibly stuff about automated wire transfers (so-called "high value payments" in payment-geek parlance).

For those of you who are just coming into the Finance space, I found this site to be very helpful in getting your feet wet [2].

Anyone have other resources they've found useful/informative?

[1] http://en.wikipedia.org/wiki/ISO_20022

[2] http://www.accountingcoach.com

http://www.scribd.com/doc/215642587/Finance-and-the-Future-B...

If you want to learn about accounting from fundamentals up, I recommend Frank Wood's Business Accounting 1: http://www.amazon.co.uk/gp/aw/d/0273712128?pc_redir=13981155...

For more advanced topics, the 3rd party study guides for the major accounting qualifications are pretty good. Try searching Amazon for 'BPP CIMA'.

Note: these are resources focused on accounting. If you are interested in finance, then you could do worse than starting with Brealey & Myers: http://www.amazon.co.uk/dp/B00DDML9U2

ACH libraries have been around for 20 years... Hope you didnt recreate the wheel...

Any good ones you'd recommend for major languages?

Technically... You cant initiate an ACH transaction. You request it from a member institution, who authorize and initiate it. Many large institutions have their own APIs... But cybersource http://www.cybersource.com/developers/ has most of what you'd need

Interesting - as I understand the process is generally to send an ACH file to a bank's file transfer point once agreements are signed and credentials/keys are shared.

Are there any libraries to create the ACH file itself from basic information like ODFI, etc?

There are a few good ones on GitHub: https://github.com/jm81/ach At my old company, we manually had someone use ACH Tools to build out these files, upload to the bank's interface, etc.: http://www.achtools.com/HomePage.aspx#!/

...if only financial institutions actually adhered to the NACHA specification :(

Right. 94 character, fixed width. What could go wrong there?

What's that, you want line endings?

I've seen \r, \n, and \r+\n. Nothing like 96 characters in a 94 character format.

Why the were you voted down?

Sarcasm, I think.

https://news.ycombinator.com/item?id=7619287

Good start. Would love to see what an actual ACH file looks like. Googling returns some specs and the like, but not sure if any contain the exact doc that gets FTP'd to the Fed.

If you google for an example "nacha file" instead of an "ach file" you should be able to find what you're looking for. Here is one example of what one looks like:

http://www.treasurysoftware.com/ach.html

"moves electronically through the banking system today"

I assume only in United States? I wonder if one can automate SWIFT transfers the way one does ACH, that would be epic.

Seems very logical, not much to learn in part 1! Does the ACH still hit the federal reserve if the two accounts are at the same bank?

Maybe, depends on the bank. It certainly doesn't have to if the bank is optimally processing the file.

there is a lot of discussions and actions being made for intra-day settlements.

Yep, here are the current details:

https://www.nacha.org/news/nacha-leads-industry-toward-ubiqu...

IMO - this is the only way ACH can remain competitive with Card transactions. Intra-day settlements and returns

Speeding up returns will be a nice bonus, though there's still a long window for fraudulent returns that can really bite.

The whole payment space is getting really interesting in the last couple of years. Banks and retailers trying to work around credit card processing fees as they became larger part of their balance sheets.

So they use FTP and files as an RPC mechanism. What happens when you accidentally overwrite files? How does sequencing happen? Do they use timestamps on the files?

It sorta, kinda makes sense and I can't really deny the simplicity of the whole scheme.

I'm sure every bank has their own protocol, but ours requires us to datestamp and incrementally sequence our filenames.

Generally, there's a confirmation step for transaction counts and file totals.

Also, there's something of a convention to never send the same file name twice.

I too was in finance. Had to write ACH processing for loan payments and collection. No fun. However, there are plenty of docs from the Fed. No test systems tho :(. I think I'll go look for some of my old ACH code just for fun.

Is there no getting around debiting party A into your own account, then crediting party B from your own account?

Is it not possible to initiate a direct payment from A to B?

I hope you guys continue with this series and make the next one more in-depth. I am very curious how everything work, especially the fees.

I've integrated with several domestic ACH APIs, and none of them are FTP based. Most are SOAP services.

What are the fees?

Random question: is this how Dwolla does its transactions--through ACH?

Obligatory "Bitcoin will disrupt this" post.

Not in our lifetime. Cryptocurrencies will just be another bolt on and thing we need to integrate. The upshot is that the banks probably won't manage this. The downside is that the banks probably won't manage this.

THIS is why Bitcoin. THIS.

seems logical

"documentation explaining the ACH system is targeted towards bankers, not software developers."

The ACH system itself is designed for the benefit of bankers, not developers, or even depositors.

Bitcoin alleviates the need for the excesses of the described system.

Bitcoin doesn't let me withdraw US $ from one US bank account and deposit it in another US bank account in bulk.

Bitcoin lets me transfer bitcoin around - which is fantastic and wonderful - but it doesn't address the problem at hand.

You don't understand. Once you're in the Bitcoin ecosystem, you don't necessarily need to backtrack out into US $ anymore. I buy almost everything with Bitcoin, and shop at Bitcoin-friendly stores. Except for grocery stores. I don't know of any grocery stores yet that accept Bitcoin in my area, but they will surely arrive someday.

Reply on news.ycombinator.com