Hacker News Reader

Comment by pc86 | Hacker News Reader

I was just complaining to my wife about this earlier this morning (we are in the US).

I made a credit card payment last Thursday (1/14) in the mid morning, maybe 9:45 AM. Ideally this should happen pretty quickly, but I know in the US the infrastructure just doesn't match my current expectations. Seemingly there was zero movement until my credit account was credited the payment Friday afternoon, but the money was still in my checking account (no holds or pending transactions).

The weekend goes by, no movement. Monday (a bank holiday) comes and goes, as does Tuesday. I get a text message early this morning (~6:45 AM) that there was a large withdrawal from my checking account - the payment showed up! Nearly three full business days before the money was actually moved, and as of right now it's still a pending transaction.

The cynic in my wants to think that it's so people who don't pay attention to their finances are more likely to overdraw/double-spend the money, but part of me thinks it's just because the US infrastructure around banking and payments is so old it just isn't capable of anything approaching real time transactions. I would love to see a logistical/technical explanation of why it takes so long for this type of thing in the US.

View on HN

Which credit card company?

The fastest possible resolution for your fact pattern would have been "money withdrawn on Friday" unless one or both of your banks decided to covertly extend you credit. This is because ACH pulls are an overnight batch process in the US.

Given that it took a few days, it's possible that your credit card company is using an intermediary to process ACH payments (there could be a variety of reasons for this) and, as a cost-savings measure, either the CC told the intermediary "We'll take your cheaper 3 business days option in preference to next day" or the intermediary told your bank "We'll take your cheaper 3 business days option in preference to next day."

The cynic in you should probably not worry overmuch about why a bank is willing to grant you free float. It's pretty pedestrian: back office optimizations, yay.

Chase card, M&T Bank.

I did know that ACH stuff is a daily batch, but I hadn't thought about an intermediary between the card and bank, that's a good point. And even without it, I would imagine that even an entity the size of Chase probably gets a discount on 3-day v. next-day.

I think Chase has problems in this area. AMEX seems to clear payments overnight for me. Chase will take days.

> The fastest possible resolution for your fact pattern would have been "money withdrawn on Friday" unless one or both of your banks decided to covertly extend you credit. This is because ACH pulls are an overnight batch process in the US.

I don't think this is true. There is the NACHA same-day ACH initiative [0] which is opt-in and at least some institutions have already opted-in.

I've experienced frequent same-day resolution of ACH transactions when using bank services from two very large US banks (one based on the west coast, one based in the east coast) and also a small employer credit union.

It's true that my experience is not common and it often does take longer than a day. Many banks still do use overnight ACH. But I don't agree with the statement that overnight is "the fastest possible resolution" -- some banks have opted-in to doing faster and have already begun doing so.

[0] https://www.nacha.org/content/same-day-ach

I worked at a data center for a bank from 1997-2001. Processing ACH transactions for our data center involved a 1970s-era NCR mainframe, a decade-old MS-DOS PC (they tried to upgrade it once, but the expansion card that handled the encryption wouldn't work with a faster system and a new card cost upwards of $10k), 9-track vacuum-fed tape drives, manually dialing a phone and then switching it over to a 9600 baud modem, and running batch-processing jobs which took greatly varying amounts of time to run. I did not handle the sending of ACH data back to the ACH system, only receiving it and processing it. I worked from 4PM "until it was done". Most nights this meant around 9:30PM. However, in situations like you described, where there is a bank holiday on a Monday, things would always take longer (higher volume of transactions, nothing was done on weekends so everything just accumulated). In situations where it was the end of a month AND a bank holiday on a Monday? Holy crap. I wouldn't get out of there until 4AM usually in those situations.

There was a great deal of hand-holding and operator intervention during the processing. The systems used were older than many of the people working on them. When software updates needed to be made, companies had to locate retired software engineers that originally made the systems and pay them astronomical amounts to either perform the modifications or consult on such modifications. When hardware broke down, like our high-speed dot matrix printer that printed on that green-bar paper with holes on each side, they had to pay specialized repair techs $400+/hr (starting from when they got in their car in a city 1.5hr away) to come out to troubleshoot. Then they had to track down replacement parts.

When you work with a bank, if something still functions, or the cost of repairing or updating it costs less than purchasing and transitioning to a new system, it does not get replaced. Systems ossify and live pretty much forever. The mainframe we used was literally the only one of its kind still in operation in the entire country (we found this out when the other people who used one, a restaurant, decommissioned theirs and gave it to us for free if we would hail it away... we finally got a disaster recovery solution that way).

I get why they don't want to migrate, but wouldn't it be sensible to create a new bank, with new systems, and gradually move people over? All new accounts go to the new system, and over 20 years or more it's migrated.

In the US, ACH is moving towards same-day clearing in two batch windows. This is supposedly rolling out starting in Sep 2016.

https://www.nacha.org/rules/same-day-ach-moving-payments-fas...

Yes, but in 2016 it'll be same-day ACH Credit only. That doesn't solve the problem. The ACH delay is on the Debit side (2 working days ) which will not be same-day until 2017 and the banks are not mandated to update balances same day until 2018. So same day ACH will not be effective till mid 2018

Per the linked article, effective Sept 23, 2016:

>> Virtually all types of ACH payments, including both credits and debits, will be eligible for same-day processing. Only international transactions (IATs) and high-value transactions above $25,000 will not be eligible. Eligible transactions account for approximately 99 percent of current ACH Network volume.

Also of note, same-day is not real-time...it is disappointing that we are not pursuing real-time transactions.

a large problem and reason why ACH's are so slow, revolve around the lack of real time fund validation.

it doesnt exist. Returns (NSF's) create a huge issue and lag in the entire process

I've asked bankers at the branch about this before and received a vague answer about how they need time to verify things, but I strongly suspect they were just non-technical and didn't know.

Is part of the problem inertia? What I mean is if there are multiple pending transactions on an account, is it even possible from a technical perspective to say "Is there $100 available in this account for immediate withdrawal?" I would suspect it is possible, but I also wouldn't expect the majority of US banking infrastructure to run on AS400s.

Isn't that the available balance? At least here in Portugal, we have three balances: accounting (based on all non-pending transactions), authorized (accounting plus all pending transactions) and available (like authorized, but just with the pending debits). The latter is the amount actually "yours", which you can use without paying interest on.

Yes but particularly for restaurants where us Americans will tip 99% of the time, the "pending" amount is often different from the finalized amount. You won't usually see that until the item actually posts, which could be several days.

I don't know how current it is but back in the 90's when I was dealing with technical issues with payments at UBS (UBS was a big Sun purchaser) one of their bankers described bank transfers as a giant exercise in "liability hot potato". Which is to say that at each step of the way the person receiving the funds was trying to avoid becoming liable for them until they were really certain they had them in an account somewhere, and people sending funds wanted to be absolutely sure the destination had accepted liability for the funds before they were willing to release them. Because when it went wrong, they ended up liable for funds they no longer had. And he suggested that there were people that tried to make that true (basically stealing from the system) and that it happened enough to be priced into the transaction fees.

The amount of money that was flowing on a daily business through the bank was staggering.

Given the crowd on HN I'm surprised there isn't at least one person chiming in that works on these backend bank systems.

A wild guess would be that roughly 90% of all systems handling this are written in COBOL running on z/OS or IBM i. And that most of the developers are currently close to 60, spending their free time with their grandchildren and not on Hacker News.

I'm not much help either since I work with mortgages, and having it all explained during a lunch break might be a bit much. There are also, depending on company and country, rules and laws about what you can talk about.

There are still a lot of developers for these systems. Updates happen constantly and developers are constantly in need and training-up.

For example, credit card legal and security compliance updates happen a lot with really strict deadlines that have to be adhered to, or else the credit card issuer gets into hot water.

Some organisations choose to merge old features that differ by country get merged in a central trunk with flagable switches. The code bases are very active.

My company (not going to say the name here) employ 500+ mainframe engineers, some working on new projects and codebases because mainframe is the preferred platform.

If you or anyone else would like to get in contact regarding COBOL or Mainframe in general, especially getting work done, especially in finance, please find my email in my profile.

> There are still a lot of developers for these systems.

I hear Initech is hiring.

I've worked on those systems and there's not much more to say, the hot potato analogy is pretty much a perfect description.

What protocols and software stacks do they use? How are they networked? What are the steps in the verification process? What prevents it from being able to work in real time? Where is this type of information formally recorded? I had a hard time turning it up. For such a critical and prolific infrastructure it's unusually opaque.

> What protocols and software stacks do they use?

Heterogenous, differs between institutions, differs within institutions, also many/most institutions are a mix of incompatible systems because the business institutions are a merger of mergers with acquisitions of mergers, all while requiring to keep constant operations, full archives of all past deals of all those legacy institutions and compatibility.

In some spaces, Java is popular, in some spaces COBOL is used, and many have something from a multitude of different exotic niche platforms. In any case you'd expect to have a mix of systems/platforms, some of them modern, some of them older than you are. You'd have 'common' platforms/stacks e.g. oracle on linux, or mssql on windows, you might encounter something also running on LAMP somewhere, and you might have e.g. a custom platform with integrated DB-like and programming-like functionality running on a particular version of AIX, all of those in a single institution.

> How are they networked?

All kinds of things. Some use common TLS channels. Some parts of business may even use simple file transfer or public email without expecting the channel to provide any security, but rely on each separate message being securely encrypted and signed. For anyone that you "don't know personally", there's SWIFT that will give you secure though not cheap messaging - if you get a swift message from e.g. FBNINGLA then you can be sure that yes, this actually is an authorised representative of the First Bank of Nigeria.

> What are the steps in the verification process?

Depends on your pre-existing relationship - if you trust them up to some limit (equivalent to a credit line up to that limit), then you do technical verification that the request is authorised and authentic and that's it. If you don't, you wait until they somehow hand over the 'cover funds' to someplace that you securely control, verify that it is received there (e.g. account statement from that place at the end of their business day), and then proceed further.

> What prevents it from being able to work in real time?

I believe mainly because multiple independent parts of the process that have no motivation to change, and often have motivation to not change - if you agree to do your part of the things faster and cheaper, it adds cost and decreases fee revenue; and changing things requires cooperation and investment from a majority of players.

Up to some part, there may be legal obstacles that prevent some particular ways to be done faster (the liability 'hot potato' issues) but IMHO those would be quickly solved by the industry, lobbyists and involved lawmakers if the motivation was there.

> Where is this type of information formally recorded?

It's heterogenous, so it's a multitude of similar-and-different things. For any particular payment system you'll have short public descriptions and also available training courses, technical standards, etc though often at a significant fee. It is niche expertise required within that niche, and not particularly PR-useful to publicize in detail, probably entirely the other way around.

I once worked with an ACH system where the transactions were in a txt file on a Windows box that was, IIRC, copied over ZMODEM after a dialup modem connection was opened to whatever the other party was (bank? processor? I don't know). Obviously, given the liability involved, I never messed around with it. But I always wondered what would happen if some amount was different than expected. One would hope there's all kinds of verification going on, but I kind of doubt it.

This was only 10 years ago.

When I worked at a data center for a bank, we had basically the exact setup you are describing. Part of my responsibilities was to take note of the amount that came through via ACH on that Windows machine (Windows 3.11 actually... this was in 2001) and enter it into a Lotus 1-2-3 spreadsheet, along with totals which I got from reports that ran after the days batch processing completed, and make sure they balanced. If they didn't balance, I had to go hunting and pray that I simply transposed a number or typoed something along the way... if I did not, I honestly don't know what happened. I had to call one of the other employees to come in and orchestrate getting it fixed. (I was alone at the data center, which consisted of a mainframe and a handful of PCs... probably an order of magnitude smaller than what you would picture if I just say 'data center') Luckily, it happened very very rarely. Maybe a handful of times in the 4 years I worked there.

potatoes hardly stay hot that long, isn't it

Huh? Would a move to a real time gross settlement system fix all of this? We run all interbank transactions over a real time gross settlement system. There is no counter party risk. Everything is backed by a giro account at the central bank.

The "need to verify" is not a technical need and can't be solved with technical means - it's all about counterparty risk, it is a business need to verify so that you're not vulnerable to all kinds of exploits and fluctuations.

It is trivial to have the payment messages happen in near-real time. However, to actually make a final credit in the receiving account and allow the end user free access to withdraw that money, you need one of three things:

1) the receiving institution simply trusts that they'll receive the money and releases the money as soon as they can - which is not realistic due to counterparty risk, fraud/revocation risk, etc and can happen only either if the whole payment happens in a single central system/institution (e.g. something like paypal) or the system includes very high fees to cover those risks.

2) each sending institution has a large binding collateral and there are appropriate safeguards that in the case of someone "going bad" the receivers are going to get paid anyway - which requires a strong central system to manage it, and the institutions deffering a lot of power and money to it (in this context, a million doesn't come close to a lot of money). This is the system described in the original article - the payment gets credited/finished first, and covered/settled between the banks afterwards.

3) in the absence of strong guarantees or blind trust, everyone in a payment chain pays the money forward only after they have been actually paid to cover the payment, adding days and days to the processing time. This is the current US system and also how general arbitrary international payments can be settled - particularly in international payments, you can get the full payment data nearly immediately, but you generally wouldn't receive the funds until every institution involved has covered their risks.

>It is trivial to have the payment messages happen in near-real time. [...] receiving institution simply trusts that they'll receive the money and releases the money as soon as they can //

Surely the message is "the money", it's not like a truck pulls up at the bank each night with the transaction balance for the day?

I don't see how there's a risk if there's a transactional process: for example Bank A sends request for money from B (eg to fulfill a payment by debit card); B reserves the money and sends a verified payment (crypto signed and what-have-you); A processes the payment and increments the relevant account with reserved funds and sends a receipt to B; B processes the receipt completes the decrease on the relevant account using the previously reserved fund and acknowledges the receipt; A receives the acknowledgement and verifies the funds as available in the account.

So, if B doesn't have funds to reserve then the payment can't be processed and A won't acknowledge any receipts claiming the transfer of monies. "A" thus won't increase a balance for any accounts. Where's the risk?

"The money" is money held in the bank's account at the Federal Reserve, and sending an electronic message to the Fed to transfer this money immediately (FedWire) counts as actually moving money. The only problem is that all banks have decided to charge $25 for this service, despite it costing them on the order of $1.

But they have agreements with banks to process the final transaction at the close of day [don't they??] - so the reserve would then charge them $1 per day per bank they had to settle a debt with? Presumably the transaction receipts and acknowledgements are binding meaning they can't legally screw another bank out of that money. So the risk is that they'll have to sue another bank in to the ground and make oodles of money if that bank refuses to settle with them at the end of the day? [Off-shore is obviously going to be different.]

Can someone speak to the situation in the UK in this respect too, please.

Insolvency screws another bank out of that money no matter what legally binding acknowledgements you have, and you won't be able to recover much from suing because, well, they don't have enough to pay.

They are rare, but not that rare, especially whenever the economic cycle again and again reaches some hard times. for example http://www.statista.com/statistics/276231/bank-insolvencies-... lists a few hundred insolvent institutions just in the last dozen years.

First, let's not look at credit card payments which work very differently from what you describe (the 'network' i.e. MasterCard company, Visa, etc play a big role) but at payer-initiated transfers.

Second, keep in mind that in almost all cases you won't have a bilateral relationship, this is unrealistic to have every institution to handle close relationships with each other - instead, usually the best case you can rely on is a "star" scheme where any A in some market can reach any B in the same market by sending a payment message + money to some "central" C and asking that C to forward the same money to B; and in more distant situations you can easily get 2,3,4 or more intermediaries for various reasons.

Third, be very careful with terms like "the account" - unless you add more specifics, all that "the account" generally means is "our accounting department believes that this is how much you owe us"; it should reflect 'having money' if accurate but it is not the same e.g. if you're unable to collect what your accounts say you own. If you say "A has $10 in their account with me" it literally means "As far as I know&have accounted for, I owe $10 to A, and should allow them to do stuff worth $10 as they order".

But for the main points - your proposal has two weak spots. 1) It requires any paying institution to have large available reserve funds with all receiving institutions. I.e. if I'm bank A, and I want my large corporate customer to be able to send $100m to any of the banks B,C,D,E,F,G,H,I,J,K when chey choose to, then I would have to keep $100m 'in my account' with each of them (or alternatively, have a credit line, where I'm sure that they will allow me to overdraft $100m) - in this case this requires having $1b of capital just so I can have this capability even in this small toy example. If you'd want to have more potential recipients instead of 10, you'd need even more capital. Part of this is actually done so in practice, but as you can see it's not that simple.

2) Your proposed messages (which are similar to real world implementation) + appropriate legal agreements only achieve that after all this ritual is completed, there is a legally binding situation that yes, the sending bank does owe the receiving/executing bank the required amount.

However, from a risk-averse viewpoint "the money" is transferred from A to B if and only if B is ensured to have full access to the full money (as in, they can actually request it driven by a truck to them and be sure that this will happen) even if e.g. A declares insolvency after sending all the required messages.

Do note that in the usual insolvency procedures, if A has $10 as "funds available in the account of A" (i.e. you owe $10 to A) and also A legally owes you $10 for the payment you did an hour ago, then you will not be able to take that $10 - all their assets (e.g. money in that account) will be split among all the creditors, so you can expect to actually not get most of that money in such an event. This is part of the 'liability hot potato'; it goes a bit deeper but you'd need a finance lawyer to explain it in more detail.

You would expect to be safe from counterparty risk only after 'your account' with the sending bank shows 0, they have legally acknowledged so (which for various legal reasons depending on country may require an end of business day, not just a signed message from them), and you have "got the money" in some form that's more trustworthy than simply 100% legally&cryptographically secure acknowledgement that A owes you the money - e.g. a message from a truly trusted institution (e.g. your central bank) that they owe you this money and it will stay so even if A goes insolvent.

These are real problems. However they are solved elegantly by central banking, and indeed that is what we have in the US: ACH and FedWire. Each bank holds an account with the Federal Reserve, and can make cheap instantaneous or free nightly batch transfers between its Fed account and other banks' Fed accounts. The strong central system extracting guarantees and ensuring minimal abuse is a major production, but we still use it for all electronic transfers.

The problem is that many American banks also process their own transfers in a nightly batch on the same 60s mainframe application they've been using since mainframes were cool.

First business night: money is taken out of your account.

Second business night: money is moved between banks via the Fed.

Third business night: money from the Fed is credited to recipient account.

If both banks were to actually do online transaction processing (like MC/Visa) then you'd only wait for the Fed's nightly batch. Some institutions like Alliant Credit Union have the rare differentiator of actually transacting with you and the Fed on the same day, instead of in separate nightly batches.

FedWire is actually instantaneous settlement, but the banks have all apparently conspired to charge an enormous markup on it. It doesn't help that cheap ubiquitous wires would outright destroy MC/Visa, who effectively get to tax all consumer spending by being the only system with live/online transaction processing for authorizations.

The EU solution to a very similar problem that we had earlier was essentially a single item in legislation (the payment services directive) that states that if a consumers payment is not credited to the beneficiary within a specific time limit, then the consumer is entitled to compensation from their (sending) institution. The industry had a couple of years before it came into force, by that time everybody implemented it.

I believe that a similar approach is the only way that can be effective in achieving a similar result in USA - unless some regulator draws a line and says "your product must meet a certain quality level or it's not fit to be sold" the current market has no reason to change, it's more profitable as it is now.

Hey PeterisP, what does the EU payment servies directive specify as max send time and for what cases, can you give more exact reference? Thanks.

The key point for this is #69 of http://eur-lex.europa.eu/legal-content/EN/ALL/?uri=CELEX:320... stating that they must be credited to the beneficiary "at the latest by the end of the next business day".

+1 business day if made in paper, and cut-off times apply - if you make an order at late evening, the bank can treat it as received in the following business day.

"for what cases" is a bit tricky. It must apply to all intra-EU deals in EUR and for the EU countries that don't use EUR, their national transactions. But for other combinations of countries and currencies (e.g. to non-EU places like Switzerland, local transfers in USD, etc), it varies, and there's some room for interpretation by member states.

>It doesn't help that cheap ubiquitous wires would outright destroy MC/Visa, who effectively get to tax all consumer spending by being the only system with live/online transaction processing for authorizations.

Cheap (actually: free) ubiquitous wires are a reality in the UK as described or in the Euro zone, and I think MC/Visa do just fine there.

I was thinking that they added this delay so that they could profit from everyones money for a few days, continuously.

This used to be a big thing, but since Check 21 and truncation that amount of money banks made has been severely marginalized. The big reason stems for risk.

The 2-3 allows the receiving FI to mitigate, IIRC, around 70% of the risk associated with the most popular types of rejects. (e.g. insufficient funds)

Flavors of fast (https://www.fisglobal.com/-/media/FISGlobal/Files/Report/Fla...) is a fun read.

In addition to the time needed for ACH, some or most of the lag may be due to fraud/money laundering concerns, even if it's low-risk. If that's true then it's a regulatory issue.

Most likely your "available balance" was reduced and everything worked fine.

"The cynic in my wants to think that it's so people who don't pay attention to their finances are more likely to overdraw/double-spend the money"

The cynic in you is right. There have been numerous lawsuits that have won against banks for structuring the movement of money so that it is the most likely to incur overdraft fees, which are a $1 billion money maker just for the top three...

Bankers are more the enemy of the people than terrorists.

It's possible to "game" this a bit to squeeze an extra $50 or so out of an empty account. Of course, it's a losing proposition given the overdraft fees, but I managed to get a few desperately needed bottles of booze this way back when I was struggling with alcoholism.

Reply on news.ycombinator.com