Tell HN: Wells Fargo completely offline
Wells Fargo is completely offline. How does this happen? Been this way since 6am PST!
And if you did take such a job, how long do you think it would take you to get the budget and approvals from all the auditors necessary to fix everything?
How much would they have to pay you to take that job, knowing how frustrating it would be to get anything done?
And now you see why banks have such terrible IT.
(One of my mentors actually works at BofA, and says he only does it because he gets to work 6 hours a day, gets a VP title and a ton of money, but nothing ever gets done)
If the answer to that challenge was one of "too hard", "it works for now" or "too expensive" that's still a strategic failure that impacts their core business now.
This is why sometimes absurd amounts of money are paid to keep the old system going as it is. It's "safer" than to move to a new one with the associated risks and teething issues.
This is not really related to costs. It's related to culture. Costs and quality of software, including infrastructure, range by orders of magnitude per theoretical quanta of software.
We don't really know how to fix this.
The only people who think auditors are bad are the ones trying to pull something an auditor would catch.
It's not just infrastructure. Quality, innovation, speed, efficiency, etc. To whatever degree you believe the old 10x programmer meme, believe this: Change that constant 10 to any value for many employees at once at a company and the impact is staggering.
Good people don't think to they are too good for these types of companies. They think the companies are too constraining and place a limit on their ability to realize dreams or their potential to perform.
They don't prefer to work with less passionate peers. Not because such are lesser human beings to be avoided. Rather, it gives them another edge. Constant, motivating, cross-reinforcement during discussion, doesn't just inspire it creates action.
Finally, most of these companies don't have tech as their core business. That correlates very directly to the fact that the most senior and influential people in the company will care less about you, being willing to partner on ideas, etc. It's not personal, Banks think bank stuff is most important and anything else is less important, even if it's a critical operational component of the business.
You're the best engineer willing to work in a black box, a niche, in isolation from your peers with constant pushback.
https://np.reddit.com/r/sysadmin/comments/ao4g2y/wells_fargo...
From what I understand, using water/dry pipe isn't unheard of. Some prefer it over Halon - https://blog.equinix.com/blog/2014/03/26/we-must-protect-thi... .
Did not know Halon could destroy circuit cards. Apparently it also damages the Ozone.
Yes. One place I worked (not a tech company, but with tons of electronics), when the fire alarms went off we had xx seconds (I don't remember the number) to get out of the building before something called Intergen was vented into the room to somehow suck all of the oxygen out, and if we were still inside we'd be dead.
It must be pretty serious stuff, because we'd have evacuation drills twice a year.
I think the concern likely comes from:
-Folks that are of poor cardiac function are going to be evacuating, meaning increased cardiac demand under stress, while being somewhat oxygen-starved. This could tip some folks into an acute episode that otherwise wouldn't.
-Folks that are of poor oxygen function, who are borderline hypoxemic to begin with. Think folks with chronic obstructive pulmonary disorder: about 5% of your employees aged 55-65 will have it.
You won't suddenly kill a building full of people. I'm guessing the evacuation rush is to make sure they're not liable for unnecessarily sending a couple to the hospital.
To put that in perspective, that's like being sent to the top of Pikes Peak (4.3km / 14,000') in seconds. Pilots flying that high in unpressurized aircraft are required to have oxygen masks. Most people will develop altitude sickness when rapidly subjected to that.
When you consider the potential for stress or panic in this kind of scenario, hypoxia emerges as a very real threat even for the young and healthy.
That's an important caveat, given how the danger was apparently greatly exaggerated.
If you've been told that the system will
> somehow suck all of the oxygen out, and if we were still inside we'd be dead
then what are you going to do in a real fire situation, when you're not in imminent danger but your escape route is blocked by flames? Better to brave the fire (or jump out a window) than submit to death by suffocation...
> VAXen, my children, just don't belong some places.
Suffice to say... yes. There are fire suppression systems that will pull oxygen out of the room. People are advised to leave the room before the fire suppression system takes effect.
If I were a customer, I'd use this as a sign that this company is not technically competent enough to manage my money.
I'm not in the US, though.
I would expect if WF decided your first loan met their purchasing criteria, your refinance would get the same treatment.
Pro tip: if you’re ever in that situation, demand the bank produce the deed proving ownership to a court of law.
I have a 30 year mortgage, but every 5 years I have to "renew" it. At that time, I have to renegotiate the rate for the next 5 years. As part of this negotiation, I can just switch banks if I want. Or to a private lender. Or to anyone really.
It seems weird to me that you are beholden to an entity you've never signed any contract with.
Credit unions usually have good offers.
Ideally you’re not buying a home that you can’t afford if rates go up too much.
Typically the Bank of Canada sets their prime rate, some time later the big banks set their own prime rates based on that, then the mortgage rates are set based on that. The Bank of Canada prime rate only moves by .25% or .5% at a time.
If you have a variable rate mortgage and the rates change, they will be immediately reflected your mortgage. This isn't as bad as it sounds - your payment will stay the same, the rate change just affects how much goes to interest vs principal. The mortgage documents will include the 'trigger rate' which is how high interest rates need to get before your payments no longer cover the interest. This is the point where you're in trouble.
For some variable rate loans, like an auto loan, an increasing rate just means that the term of the loan gets longer or shorter.
As always, ask questions. The bank, in Canada at least, doesn't really want you to default on the loan. Ask about the trigger rate, ask what happens if it gets hit, ask what happens if rates go up but don't hit the trigger rate, ask about lump sum payments.
Story time: several years ago I took out a 10 year fixed rate of 2.99%. My thinking was that since the base rate couldn't really go down any further, I was locking in a good rate.
As it's turned out, so far I could have had a series of 2 year fixed at around 2%, so this was potentially the wrong move, although the maximum downside was limited.
My parents on the other hand took out a 12.99% fix in the early 90's, which turned out to be incredibly unlucky given the unprecedented low inflation of the nineties and noughties.
Source: https://oag.ca.gov/news/press-releases/attorney-general-bece...
https://finance.yahoo.com/news/every-wells-fargo-consumer-sc...
Tangentially related, I highly recommend the movie "Out of the Clear Blue Sky." Cantor Fitzgerald was a bond trading firm at the top of one of the twin towers and lost every employee who was in the office on 9/11. Incredibly, despite losing the majority of their employees and despite losing almost all of their trading infrastructure, they managed to resume operations in time for the bond market's reopening 48 hours later.
And then starting around 2010 but rapidly accelerating around 2014, everything about them went to shit.
The best explanation I can think of is that John Stumpf is a slash & burn sociopath, juicing the numbers so he can get his 473x-the-median-worker paycheck while ruining the company. He wouldn't be alone in the financial world, but it's a shame that a 150+ year old institution can so rapidly go down the toilet.
Except then PayPal made a few more attempts for god knows why and each time Wells Fargo kicked an NSF fee our way.
Now, PayPal shouldn’t have repeatedly attempted a rejected charge. But, Wells Fargo shouldn’t have allowed those attempts. They just couldn’t help themselves to that $35 NSF fee though.
We fought it to no avail. With all the NSF fees and interest (and fees they added to fees while we fought it), what started as a $300 transaction ultimately cost us over $1200.
Wells Fargo is now and was in 2009/2010 a criminal enterprise.
All the crazy sales numbers and bogus account shenanigans were going on back in 2003/2004 when I worked there. I ratted out more than one professional banker to branch managers and up over that crap. A fun one was the home equity lines people would open without customer knowledge and link up to overdraft protection. The customer would never owe, nor know, anything until one day an overdraft hit their equity line, and then they got notified of late payments.. I don't miss working for a Bank.
The WF business is clearly set up to confuse and exploit consumers. My credit union websites have always helped me do what I want and need with my money. This includes the tiny local credit union in Idaho.
Im employed by WF and even Im a little bewildered by the fact there doesn't seem to be any redundancy implemented somewhere.
You can't really roll back say 10 minutes of transactions, so are you maintaining 2 parallel systems? How do you keep them perfectly in sync?
This isn't my area of expertise by a long shot, but it occurs to me this is probably hard, especially when your codebase started in the 60s, and has been accreting ever since.
The backup doesn’t need to be in the same exact state as the primary (it’s not meant to service requests), it just needs to have a persistent log of what changes were applied so that it can roll forward when needed.
Most relational DBs do something like this for their DR product offering. Oracle has Active Data Guard. DB2 has HADR.
(Even if you can just put anything in the email form)
And if anyone ever figures out that isn't the way it is, and that the numbers are not representative of anything of substance? If nations refuse to honor the claimed 'transfers' done through these rickety electronic systems? It would make for an interesting few days.
I mean truthfully when do you get to test your redundancy against a true disaster. It was a mess. WF is 20 companies rolled into one so the fact the disparate systems from 10 different banks works at all is kind of a miracle.
Netflix designed their stuff from the ground up to fail over. Large monolith corporations who've inherited systems from other companies they've bought or merged with have challenges you won't see many places that have benefited from the 30 years of lessons that were taught at these companies.
Chaos engineering and AWS weren't real things when they started building the company. And the system they have now doesn't resemble much of it was once.
Truth of the matter is they invested more in their infrastructure, but that's because their business plan required them to grow on the back of technological advances. Banks, it's seems, do not. Or maybe they do, and the some of these start up banks will usurp them.
But I’m guessing wellsfargo just doesn’t have a reason to care.
No, it can't. Any loss of customer-facing functionality is a critical outage ("World Problem" in company terminology). There are a relatively small number of customers, but the terminal is critical to the operations of those who buy it. The terminal going down for eight hours would be a world-wide headline in the financial press.
A Tier 1 test that simulates loss of a datacenter takes a cluster one DC virtually offline. This puts an entire subset of services offline in that DC entirely. The test is coordinated with the teams who own the services to ensure their services fail over correctly. Any service disruption during the failover is a test failure. If it passes, the customers don't even know it happened. The goal is to be able to lose an entire DC and have the terminal customers not realize it until they hear about it on the news.
Do you know what Bloomberg does? It powers equities trading markets around the world, 24/7. It isn't just news.
You can bail out of a test at the first sign of trouble. When a real outage hits, there’s no telling how long it will take to recover.
I've worked for banks here in Australia. Everything is 30+ years old. It's a shambles.
Would you honestly want to go to a bank and say "if we unplug this, we can find out what fails."
You're pretty much damned if you do and damned if you don't. If you touch things that are working you could break them. If you don't touch things you never know what'll happen and you get fewer opportunities to learn. Move your servers around geographically and you might improve the odds that anything is working by reducing the odds that everything is working.
I don't think we're quite to a place yet where having servers down can be characterized as a non-event. Even if the customer can't see a behavioral difference, business units still tend to get quite anxious, and sometimes their theatrics put the whole process in jeopardy (not unlike trying to rescue a drowning man). It just hasn't been normalized yet.
If a customer can't pay a fine - can't use their bank account - they go to jail. https://www.telegraph.co.uk/finance/personalfinance/bank-acc...
These are pretty different outcomes.
I’m serious.
Are you sure you're not one?
Also, failover is hard. Few companies outside of a few larger Internetz companies can really do it well.
After all their logo is that of a stage coach aka the wild Wild West(robbers, thieves, etc). They do not hide who they really are.
Example of high-level guidelines (Singapore): http://www.mas.gov.sg/~/media/resource/legislation_guideline...
I think that in Switzerland all major banks test their disaster-readiness (by switching everything to their secondary datacenters & working locations) of all critical applications/software-layers and employees at least once every 3 years - reaction/recovery times depend on the criticality of the service provided by the person/application.
Big companies tend to defer risk. Managers and project leads want to start new projects rather than upgrade existing infrastructure. Combine these forces and sometimes you get a catastrophe.
Wonder if it was the switchgear that failed. Amazon uses custom firmware in its switchgears because this happens so often (Superbowl 2013 etc.)
https://perspectives.mvdirona.com/2013/02/the-power-failure-...
https://perspectives.mvdirona.com/2017/04/at-scale-rare-even...
https://perspectives.mvdirona.com/2017/12/when-you-cant-affo...
great job guys
- for instance, in cali: https://www.golden1.com/
- high interest accounts: https://www.bankrate.com/banking/checking/best-checking-acco...
- good rate online bank: https://empower.me/
get better service, lower/no fees, and good convenience without all the economy-damaging selfishness and hubris.
There was no interaction with user or account data on my end, just CMS-building, letting them go crazy with coupon promotions or pet shelter PSA's or warnings about ATMs being down, etc. They'd pitch me on a new project, I'd send them a timeline, and we'd add new features or tools to the website. I even hired a local college student to create some super-basic but useful financial tools for the site.
I loved how small-town-feel the whole thing was. At the beginning of the relationship, they gave me a list of broad requirements like "SAS-70", I found a DC to match, I sent them a contract, and I couldn't believe I actually had a banking institution as a client, 4 years out of school and a brand-new business owner.
Eventually they merged with another CU and went away, but apart from the very occasional "server down" notifications while I was on vacation, those are some really fun memories.
when i choose a credit union, i optimized for two things:
- is there a branch close to work or home? this is mainly to develop a personal relationship in case i ever needed a loan. i'd deposit checks and withdraw cash in person occasionally.
- good rates on interest bearing accounts (at least 1%, but often >2% apr)
if you work for a larger institution, disney for example, joining their credit union is convenient.
* all works in an app
* totally free
* no fee when using the card abroad, exchange rate is the bank rate
* notification of the purchase right away
* it tells you where each purchases were made, on a map
* it rounds up every purchase to save them in a saving account
* you can create as many saving accounts as you want, in one click
* every purchase is automatically categorized between bills, eating out, entertainment, etc.
* you can see how much you spent in a month, and in what
* you can send money to people near by, or by using nicknames
* searching your transactions is super easy
* you can freeze/unfreeze your card in one click
* easy transfer of money
I think overall, the app has so many features and it's pretty slick.
It doesn't have no fee foreign exchanges, no rounding up, but does basically everything else there.
https://www.nerdwallet.com/blog/banking/best-high-yield-onli...
So these days there's very little physical money. Most of my "wealth" is just entries in various digital ledgers. My bank says I have $XXX and my brokerage says I have $YYY and my retirement account says I have $ZZZ.
Let's go with the bank account case. Is it possible that a catastrophic accident or attack could wipe my balance down to $0 with no way to recover? What if a data center was nuked? What if two data centers were nuked?
How much redundancy is in the system? Are there third-party agencies that track private bank ledgers? How hard is it to take them out too?
Ever since I read "The End of Alchemy" (a great book, btw) this thought has haunted me.
What would happen if there were no records? Surely there’d be lots of people making significant deposits in between snapshots. If you’re “paperless” I’m not even sure how you’d reconstruct your balance.
Would certainly be time consuming, and in this case time would certainly be money.
Bear in mind the live records are only one part of what a bank keeps.
I'm genuinely curious what the process would look like, if it even applies. Where would you even start if the bank just suddenly says "you have $0"?
But even if I did, what do you do from there? Call someone and say "I have a piece of paper that says I should have $x but they say I have $y"?
Civil claims are based on a preponderance of the evidence. if I go into court with a piece of paper from the bank saying that I have $5,000, and the bank has nothing to say that I do or don't have any money there, I have more evidence on my side than they do on theirs.
In reality it's probably not that simple.
Although I'm not sure if it could withstand a nuke, its in a mountain, so, maybe?
If you are talking about actual nuking then the story might be different. Not all backups can survive an EMP. I expect the biggest problem would be getting people to care about bringing the system back up. I think food and shelter would be of primary concern.
Please be advised that the Wells Fargo Gateway is currently unavailable.
We're experiencing system issues due to a power shutdown at one of our facilities, initiated after smoke was detected following routine maintenance. We're working to restore services as soon as possible.
We apologize for the inconvenience as we continue to work on a resolution.
We will update you no later than 3:00 pm, Eastern Time.
https://en.wikipedia.org/wiki/Hanlon%27s_razor
recall Stuxnet, its possible this was an attack, somesort of malicious mod to firmware, but_ Hanlon's razor.
from the reddit:
"throwawayfordays75 1399 points 8 hours ago*2
Throwaway since I have first hand knowledge. Fire suppression went off in one of their main Data Centres from some utility work this morning. No power to any of the network or compute equipment and some failovers did not work as expected. "
At this point im wondering what "utility work" was happening.
"everything minus core network gear was manually being unplugged from any PDUs to help the control the initial power-on."
Can't we... Why don't we have rack hardware that can handle this situation? I thought some HDD RAID solutions had circuitry to keep them from browning themselves out while spinning up the disks. I guess I'm surprised this isn't a solved problem at the rack level now.
Or have we been so focused on never cold booting a rack of servers that we haven't spent any effort on foolproofing of cold booting a rack of servers?
[Edit: answering my own question] apparently these exist and are called Managed PDUs. Can we deduce WF doesn't have them?
This does not impact Stripe customers at this time. Specifically, the Stripe API and Instant Payouts are functionally normally.
(We're aware of the issues Wells Fargo is experiencing, and have an incident response spun up internally.)
www.bloomberg.com/opinion/articles/2016-09-09/wells-fargo-opened-a-couple-million-fake-accounts
[0] https://api.twitterstat.us [1] https://help.netflix.com/en/is-netflix-down [2] https://developers.facebook.com/status/dashboard/
this is what the merry band of racist edgelords have fixated on?
Seems like it would be more /g/'s thing.
Pardon my immense ignorance!
This thread has primarily focused on redundancy and software architecture. That could be the case, but there is no better way to fight proxy war via hacking of banks. It’s a domino effect... pull your money out of the bank, the bank calls loans, then there is credit crunch, investors lose confidence in the stock, value lost etc... the enemy has secretly inflicted civilian problems across the economy. Distrupting the banks and the flow of money can lead to a revolution... or take your mind off of America’s enemy or diverting energy else where.
Oh wait...
Matt Levine wrote the other day: "...people think they don’t trust the traditional financial system, but in fact its most basic functionality—not forgetting where it put your money—is so reliable that people just keep assuming without evidence that the crypto system must work the same way.
Surely a big crypto exchange must have, like, systems, and lawyers, and more than one employee, so you can trust it, right? Wrong! If I ever buy Bitcoins I will keep them at Gemini, the Winklevoss twins’ exchange, just because I am absolutely sure there are two of them."
This stuff just keeps on happening, but crypto folks always make some excuse as to why it's not actually a systemic problem.
OK, what's your point? Nobody claimed it was perfect, just that it happens to be immune to the particular failure mode which happened here. Can you accept that that is true?
> This stuff just keeps on happening, but crypto folks always make some excuse as to why it's not actually a systemic problem.
What "stuff"? People losing coins? People lose cash all the time, does that make it a "systemic problem" with cash? They are just different technologies with different trade-offs.
It's understandable that you hate the cryptocurrency community but that doesn't mean you need to deny every possible advantage of cryptocurrencies.
What exactly do you mean by "immune to the particular failure mode"? Is bitcoin immune to server or data center outages? Perhaps if you very narrowly limit which ones you are talking about to the bitcoin network per se.
Of course! With bitcoin, nothing is the fault of bitcoin and always something else's fault. Bitcoin can never fail. It can only be failed.