Bank of England to crack down on 'secretive' cloud computing services
itnews.com.au
itnews.com.au
I would say it's nearly a 50/50 split until we have conversations about how our product actually works and the incredibly sticky problem that is PII...
Once the risks are reviewed in open and honest ways, we find that virtually all of our clients would prefer to keep our solution on-prem. Only those who have already made a full step into cloud compute have to continue to take exception for obvious practicality reasons.
I hope this kind of risk assessment becomes more common. I'm used to people not caring until things blow up on their faces.
Or, you could have a single executive action revamp IT resource acquisition policies. Simply mandate that internalized self-service resource provisioning capabilities be developed.
It is not rocket science to put a web dashboard around vmware or some other virtualization solution. Most of them already sell something like this as part of their feature set.
You could set something like this up in a week if you had enough buy-in. There are no excuses if you want to win at this kind of game. The bad guys are way more patient in aggregate.
A 2nd perspective - One of our customers has a "traditional" process for setting up new IT workloads, and we were still able to get 3 servers provisioned within 8 hours along with 3 new publicly-routable IPv4 addresses and matching DNS+TLS certs. Anything less than this in 2021 for any organization is indicative of sheer incompetence IMO. We do have some customers that are really slow, but they are also really small. I don't think any F500 is taking weeks to provision SQL Server anymore.
The people that need to plan, coordinate and install all of this are also pretty heavily overworked at the moment because, believe it or not, there is a very large shortage of skilled sys admins in the world.
Using the cloud doesn't solve those problems, but it does help reduce their impact.
As someone who has worked as a software developer for big NY banks for the past 25 years, that's simply not true.
The answer is, it's complicated. JPMorganChase for instance has a $12 billion annual IT spend. They do a LOT of different things. Admittedly certain things are best left on prem for regulatory audit points (more with respect to resilience/business continuity rather than security.) But a substantial portion of it could be moved to the cloud at some cost savings.
Additionally the brittleness of the service and database infrastructure is a pathology of the on-prem environment rather than an argument for it. Cohorts of SA's and DBA's are wasting their time doing work which in a modern environment would be scripted and more flexible.
Getting the lawyers on both sides to agree to language for the ongoing certification(s) that will also satisfy the internal auditors, the outside auditors, the regulators... suddenly something that sounded simple when the contract was signed takes several meetings and a lot of back and forth.
If you run 100% of your IT workload on-prem, the ability to control the flow of data can be boiled down into a physical exercise of following fiber channel cables in your own datacenter. Having a unified set of firewall rules that define your entire public interface also helps a lot.
You can actually make deterministic guarantees to your customers that not only your own systems are secure, but also that the systems of your vendors and other 3rd parties are as well. The moment you start configuring site-to-site VPNs with 3rd parties across which you intend to transact sensitive business knowledge, you are surrendering an entire mountain of security constraints.
If we are being honest with ourselves, a lot of shops that are 100% on-prem probably have worse security practices than AWS, et. al. Perhaps the biggest hazard is really the hybrid model. If some fintech went 100% into the cloud without even an HSM on-prem to worry about, then you could probably have a solid argument on the other side of the spectrum. Also, remember that multi-cloud might seem like a resiliency measure, but it also adds another target to your back.
The middle ground is where all the pain seems to be. Hybrid cloud usually means more required trust than most organizations ever wanted to enter into. I frequently find myself as the harbinger of bad news when I get into deep-dive technical calls with some of our customers. Turns out a lot of the other vendors we work with like to bend the truth in order to make a quick buck. Many perverse incentives are pulling these massive organizations into hilariously-contorted IT stances, and some of us are starting to see a consulting opportunity.
I suppose the infamous Equifax breach was due to a secret fiber optic cable running out of their datacenter?
No, of course not. But when you're dealing with physical infrastructure you can actually touch, it's much clearer and more certain what you're dealing with.
Even on a really simple on-prem scenario, you have switches to configure, vlans to setup, hardware drivers, a gazillion updates to make all the time and a tonne of employees making it all very difficult. The fact I can see it physically doesn't realy help me that much.
I work with some intermediate vendors in this space (they have direct access to the credit bureau data), and their security mechanisms are of concern. I am under some very strict NDA constraints, but I can say that there are serious problems and I am not surprised that breaches occur with regular frequency.
You can barely trust your own in-house developers to get these things right. How can you possibly hope to trust many other additional parties to get it right simultaneously as well?
Does it matter? You have the same freedom to fuck security up setting your AWS infrastructure as you have setting your on-prem infrastructure. All the very competent AWS staff is able to do is add less risk, they can't save you from anything.
You can make a "deterministic" guarantee, whatever that is, that your systems are secure? That's seems pretty bold and probably dangerous, no?
Its not dangerous in my experience. The more dangerous angle for me is this belief that it is impossible (or hopelessly difficult) to build a secure system.
The reality is that it is only possible if you are willing to take total ownership of the entire vertical. If you control every single byte that enters and exits your enterprise, you can prove that things are secure. Is it practical to do this in all cases? No. Is it feasible in theory and in certain cases? Absolutely.
If you buy into the 3rd party hosting game, you instantly lose control over the critical variables you would need to in order to create the opportunity for these sorts of guarantees to exist in the first place. You (and your customers) will be stuck wondering about side channel damage and human factors that you have no direct control over. When you own the hardware and the real estate it is parked on top of, you can start to reel these things back in really quickly with powerful policy frameworks (2-person rules for critical changes, mandatory checklists, etc). These sorts of policies seem to work really well for very tricky areas like keeping our nuclear weapons from doing inappropriate things.
Do you build all your own hardware from raw materials? How do you know everyone in the supply chains is perfectly secure?
Attacks have succeeded against the CIA, against RSA, Google, and many others. Nuclear weapons plans have been stolen. I would not trust a vendor who claimed they could guarantee security.
The amount of financial data that could be exploited at once is magnitudes larger in a popular cloud. I don't think it's strange that a regulator might look at that failure point with some trepidation.
And I don't expect my cleaner to service my car. Banks have DBAs, network and physical security experts on tap (just like cloud providers do).
To look at it another way, how much of the risk is about the physical location of machines, and how much is about who operates them?
I have no idea if the last 2 ("OS", hardware) is even achievable in 2021. I can always get my source code from git, and possibly the 'configuration', but can I expect to run cloudy code in 7 years time?
It's similar to HMRC's "we can come after you for unpaid tax for 7 years", but you can only go after the HMRC for 5 years for overpaid tax.
The rules are ultimately for the benefit of long term investigations such as the LIBOR rigging, and Guinness trials.
https://www.theguardian.com/business/2016/jul/04/libor-riggi...
[0] https://www.fca.org.uk/publication/finalised-guidance/fg16-5...
Call me crazy, but I would trust aws, microsoft, and google with my PII and finances before I would trust Wells, BofA, JPMC, Goldman, et al.
The cloud giants pay their engineers more and technologists are second-class citizens at the financial institutions - infer what you want from that.
Again, this stems from the culture of financial institutions where tech workers are viewed as support staff, not strategically critical, despite what financial institutions say publicly.
Visa regularly makes double the profit per employee of FAANG - so, yes, they could easily pay more.
They have profits. They just aren't in the US.
Google/FB devs total comp (including RSUs) will easily beat Goldman devs by 150-200%, depending upon your performance.
This is all with worse work/life balance, low flexibility, (GS was the first big bank to recall everyone and enforce working on site I believe.), a 'conservative' tech stack if you're lucky or a simply unpleasant one if you're not, and generally not being valued the same way as you would be at a FAANG or typical tech company.
If so, $100k plus a 15% bonus is laughably low. Even shitty startups pay new grads more than that.
My understanding is that GS's interview is actually harder to pass than a lot of FAANG companies. So you have to wonder why anyone would even interview there if the pay is that low...
Maybe you and I have a different concept of payday.
What makes GS less than FAANG is the annual stock grant FAANGs throw at software engineers, and of course it Finance.
Technology companies lean the other way and enforce security through comprehensive automation of the controls they must follow.
Many banks and other financial institutions are finding this out now in their desperate bid to find competent COBOL programmers to continuing maintaining legacy critical applications running on mainframes when most COBOL programmers are retired or dead, and more are headed that way every day. It wouldn't shock me to find out that the effects of COVID being weighted towards worse outcomes for older people had a material effect on the human resource risk of using COBOL-based critical systems.
Banks will absolutely hold on to anything to avoid an unknown risk as long as they think they can hedge or mitigate known risks, and utterly fail to acknowledge the truth of unknown unknowns. This will ultimately be their downfall if governments ever let them follow standard business outcomes, otherwise they'll eventually absorb some upstart to keep hedging forever.
Nassim Taleb has written entire books on this fact about banks. It's interesting that the same flaws in their financial thinking apply to their IT infrastructure.
There is a certain irony in this statement, given that such an act would require approval from the queen.
I very much doubt that anyone would not use the cloud because of a theoretical de-isolation bug.
Also, by the time you found out, it would probably already be too late anyway if you were a victim. If not, you just switch it off.
I put on my seat belt when I drive on the highways even though a nasty crash at 120 kph would likely kill me. Not using a seat belt because you will be severely injured anyway is not wise.
Given the amount of profit banks make what is the Downside of having them be resilient against public cloud failures?
All the concerns about vendor lock-in are ridiculous right now. Prices have consistently lowered for all cloud providers.
They are subjected to the pay scales dictated by bureaucrats and politicians. Of course, they can't attract the caliber of engineers that works for large commercial cloud providers, so they have to settle for programmers that didn't make the cut. And the leadership is non-technical and from the financial and government worlds (you know how these "elites" see programmers and software engineers in the UK...).
The existing bureaucrats running the bank probably see the Cloud as something that will reduce their headcount (why would they have sysadmins and datacenter employees on their payroll when they can simply buy it from a reliable provider), thus making them look less important to other managers. The existing unionized employees see it as a threat to their stable jobs (they are now competing with engineers that are 10x their caliber). That's a pretty bad thing for both.
They're concerned about all the major retail banks using the same cloud provider and then that provider having a major outage.
Individual banks having an outage is an issue but one that can be handled. Three or four of the main banks all going down at once could be catastrophic.
I wouldn't worry too much about that. Major outages are exceedingly rare. I can't say the same for on-prem infrastructure.
Theoretically, the Ts & Cs cover everything but clearly you cannot reasonably mitigate risk on the basis that the "supplier told me that it wouldn't happen".
You have a (sometimes legal) requirement to do due diligence and at least decide what additional controls you can have to help with compliance and what your BC/DR process is if everything really goes south.
Realistically, I don't really see AWS or Azure etc. being able to promise that their systems can never be hacked/broken. On the other hand, the assumption that doing things on-prem removes this risk is naiive since we can make just as many cock-ups even if we work for the company. Anyone ever forgotten to setup a firewall or vpn properly?
True story: When I worked at a security company, the sales team wanted to demo a digital camera-over-IP system and the IT Manager told them not to use our internal network. They ignored him and plugged it in, flooding the network with packets that took down the computers and phones for about 30 minutes until we worked out what had happened. That was inconvenient but imagine if they had plugged in something much worse "because sales"
So that you can just run a bunch of your own servers, lay over the 'Stack' and then get nice Lambdas, provisioning, and other things.
Imagine if you could just buy some hardware, and have some regular IT guys reproduce most of Amazon without the Amazon?
That would be a 'reverse revolution'.
National Regulators could also support strategic investment in the sector, i.e. 'financial ops have to be backed-up in-nation by a local provider' i.e. some kind of forced local diversity in the system by regulatory fiat. As an idea.
any other more detailed breakdown of this somewhere?
What exactly is the concern here? Cloud compute is becoming cheaper over time due to market forces. Its not like Amazon is cornering the market for CPUs.
Top of the article:
> Concentration of compute could threaten financial stability.
If BigCloud goes down, so does banking, and banking doesn't seem to like that - and I'm with them.
There have been many horror stories of businesses being banned, blocked, or messed around by Google and Amazon. Their policy does not protect anyone but themselves. It is within the realms of reality for bank to be taken down by them in a matter of days.
Our current leaders are useless in this regard. They'll be given a stern warning letter and nothing will come of it.
The cross-bank correlated technical outages and general “fear of the new (15 years now, but continually being enhanced and changed)” is harder to get beyond.
I work for near FAANG company who still sends important email information out to people. One day, many years ago, a new gmail feature landed: automatic smart tabs - and ALL of our email started landing in promotions. And those mails were send from IP addresses which were dedicated for these and only these kind of email, so we started to panic.
We started going through the correct channels for reporting this as a problem, at which point we got a "cheers, we'll get back to you in 2 weeks". At that point we were certain we were loosing money in the millions soon, so we walked over to marketing - Google PPC (pay per click) to be specific, and asked them to get hold of someone high enough at Google, we don't care how, or over what channel.
Within an hour, the change in gmail to put everything in promotions that was "noreply@" was rolled back, and our arses were saved.
If we were still in the early 2000s with countless small email servers and providers, if one of them done this, their customers would be angry at them. But since people believe Gmail is their saviour - and because nobody can get hold of anyone at google to report a problem - now it was our fault, despite the fact that we had absolutely no idea what went wrong, and there were no changes on our side.
My summary? Fuck the cloud and the centralized services; I want the small, loosely connected internet services back.
Sure, hedge your bets but I would hope that most people using cloud are smart enough to at least get it 95% cloud-agnostic so if price gouging occurs, they can leave.
Then there’s the cost: Cloud is extremely expensive over the long run; more-so than the equivalent on-prem.
The long term solution here is to add better Internet in all regions, similar to our highway system today.
Then any company of a large enough size can build out their own DCs on the cheap.
It would also inject much needed dollars into the non-coastal areas.
The BoE is the central bank; to a great extent they get to define the territory. Especially with regard to risk. Basel III and that kind of thing.
> they can literally go shopping for data center and IT companies to serve their needs.
No, it's not about what they themselves buy, it's about what all the banks in the UK buy. A situation where all the banks are using the same cloud provider(s) is unacceptable because of the high risk of correlated failure. It's bad enough when one bank goes down and its ATMs stop working; if EU-west-1 goes down and takes out all the banks, that's a disaster, and the BoE is rightly taking steps to prevent it.
If BoE does not like some risk that really does come with the territory and everybody who uses public cloud is subject to that risk, oh well, then that would imply that UK banks have to keep out of that territory and can't use public cloud for their key services - it is a bit drastic, but certainly something that BoE has the right to rule if it really wanted.
On the other hand, if there's a way to use public cloud and meet the requirements, but current major vendors are simply not offering it, then BoE can make up arbitrary requirements, followed by the major UK banks going to the major cloud vendors with a proposal "meet these requirements or we won't use your services, because we'd be prohibited to".
Even if AWS itself is more reliable than every bank's on-prem solution, that's no good if it goes down for everyone simultaneously and I can't just use a backup credit card.
They're likely only looking for ways to mitigate these risks, not necessarily a complete return to self-hosting.