Don't build private clouds
m.subbu.org
m.subbu.org
Once you get above a few boxes though, you might need some pricey geeks for a month or two to rig up cobbler, ceph, openstack, proxmox, or whatever-takes-ya simply because it's quicker than getting up on all that yourself right away -- but really there is no magic involved here and it's just so you can have it totally manageable as you grow.
The tooling is much more friendly these days; and to be totally honest -- provisioning probably isn't the problem you're trying to solve anyway (more likely packaging/deployment/ci is the pain after you got a bit of breathing space on perf). I've seen loads of folks coming back on prem spend months rigging up crazy vm provisioning things when their estate is basically static because they didn't realise they were solving the wrong problem.
If your ops time is spent building machines, then sort that out. Probably and more likely it's spent on getting apps running and dealing with moving traffic about and deployments and such; so that's the real thing to work on.
I think the 'complexity' argument seem so loud on HN as the opsy people are the minority here and it's not exactly a sexy/exciting/fun thing to pick up, and when it goes wrong it's brutal, especially when you can be nailing features instead.
All I'd say is -- regardless of what you're doing, getting into containers at this point makes your life quite a lot easier; you get a standard deployable unit of code you can throw into aws, or on prem depending on your workload, and it removes a lot of the anti-patters of config mgt.
It's been a while so I don't remember the specifics, but we did eventually have some but not all parts in the cloud.
Look for the equivalent of https://www.microsoft.com/en-us/TrustCenter/Transparency#Tra...
(Full disclosure: I work at Microsoft)
Maybe I'm too cynical these days, but I find it pretty laughable that a statement regarding 'data sovereignty' has any real meaning at all these days.
Unless you control the hardware, you have no idea what the hypervisors are doing on your rented vm's. The idea that national actors don't / wouldn't pull anything they want from them seems comically naive to me. Christ, we can barely trust our own hardware anymore with backdoored firmware crap, IME and such..
Realistically though, most companies don't care so it's not a problem if you keep the blinkers on; their compliance folks just want the document which says it's not happening and they'll never know when it does so who cares. Their ops folks can barely use the APIs probably, let alone contemplate the impact of someone reading the memory of my databases from the hypervisors.
As an individual and someone who cares for my data or that of my clients though, that's not enough for me, and I'd never trust tnese folks with any data, only mindless compute and serving public content...
Q. Don't you share this data over networks from time to time? A. Hell no.
Q. Is all your data siloed and bolted down to one place never to move anywhere A. Yes that is the plan actually.
Q. How do you trust the data to not take a hike in some random direction you don't fully trust A. Easy, it never gets out of my data center.
An IT director that works in a pharma company, will simply laugh after reading the "don't build private clouds" article
Most things are CRUD;
Current gig is a load of on-prem DBs' -- AWS boxes might hit an API in one of the datacentres which can query it for a given user, I think there's a difference between using AWS to serve HTTPS vs Hosting the whole db there (this is marketing/tracking data)
It's the difference between AWS/USGVT being able to decrypt a HTTP convo to a specifc user after we've combined a lot of data and having access to all 200TB of it..
1> Google deleted People's account who bought Pixel from reseller. They have to beg google to restore their data, contacts. Not going to happened if you have it in your own cloud.
2> Windows 10 is full of telemetry. Imagine someone knows exactly what software you used, when you used it, for how long. Advertisement are built-in with system level. You don't own your machine anymore, Windows 10 can decide when it want to upgrade itself then lock you out from using it for hours. Your data now is in Microsoft's cloud, but you don't decide how they can use it, who they are going to sell to.
For depository reason? Sure of course, cloud is reliable on up time, expandable with a few click. But if Cloud provider decided to wipe the account with whatever reason, what you are going to do?
> Erm, no.
That can come across as a rude way to disagree with someone (it does to me).
> There is no uncertainty, only lack of awareness […]
That's dismissive (it reads as “you are not uncertain about this matter, you just think you are”) and not very helpful either.
For some organizations using a private cloud is not a technical decision but instead a political/ethics one.
As an example I would bet that there are several companies out there who would not care if all services were down for a full day (in their private cloud), as long as they were certain that no government could easily access them.
(the US are not counting since this is THE government many others are not trusting and the home of the public clouds)
Just like goverments find ways to allow private companies to run their cellular networks, phone networks, tv cable networks, power grids - the same reasoning should allow private companies (public clouds) who do it much better than goverment employees, to run their compute services.
There's a lot of gov.uk stuff running on non-gvt gear these days thanks to GDS; (aws, skyscape, others on gcloud) -- the problem with how that's gone though is while they're actually running in "the cloud" and tryig to do things more 'modern' the architecture has to be accredited by a bunch of GCHQ folks who apparently are totally insane yet you can never actually talk to. And so you end up only being able to build what looks like a 2000's datacentre arch with 500 /28 vlans and weird IDS appliances and so on, otherwise CESG won't let you hook up to any of the data-sources you probably need.
In that style arch, where you can't just provision things and changing even a netmask requires your whole architecture to be re-accredited otherwise you'll lose access to the PSN; what possible benefits are there of running in the cloud?
It's more expensive, needs constant oversight from accreditors and generally sucks.
The legacy gov DCs are a shitshow too; but that's down to people like capgemini and ibm or whoever and well, fuck them.
http://www.theatlantic.com/technology/archive/2014/07/the-de...
TBH; the only people who would have agreed with me knew my points anyway so bother?
I felt like as this is my specific area, that I have a kind of responsibility to people who read misinformation like this to help see through such stuff. It's hard to do for me without getting kind of angry though; opinions like this is why I end up having to deal with 800k/month aws bills and 3 hour deploys..
Either way; regardles of me -- please, people -- if you think that the 'typical journey' of implementing your own infrastructure and migrating a legacy application to run on it is ANYTHING like the description in this blog post, then please take it from me as someone who moves horrible legacy codebases into CD for a living, that you should perhaps seek some alternative advice.
Suubu; I'm sorry, I'm not trying to be mean but your typical' private cloud journey is complete and utter horseshit.
Can you please actually elaborate, or it defeats the point of your entire post. While the rant may have helped you, I don't know you, I don't your history, and this post doesn't encourage me to go through your post history to find it.
At the same time, OP's post filled me with a slight sense of dread as I'm about to go into a private cloud environment and wondered if I was making a mistake. Your post has certainly calmed those nerves. Actually having some meat with the mustard would help more.
Thanks in advance!
Lets make some assumptions on this gig you're starting then, we'll imagine you're going into some job at a place which has a horrible legacy codebase (lets' go medieval assume struts and oracle 10, deployed with some ancient bamboo/rpm concoction which takes forever and always dies and the existing ops team barely keep the lights on, let alone are capable of keeping up with the rate dev is trying to let the business move..).
So you land. They've got a fresh budget, a load of bearded scala devs who are rewriting the whole thing as microservices and words like "ISB" and "ASYNC" are being thrown around. Hell, someone even mentioned elixir in the bar on your welcome drinkies.
You're going to drag this horrible monster into the new world, but to get there and to a better arch you're going to have to replace it a bit at a time, and while that happens the code that's making all the money needs to be maintained, and it needs to change faster. And you're going to do this on prem.
So. You land. On prem, ooooeh baby. Realise: the pain you're going to have is not going to be how you instantiate vms to run the apps or spin up loadbalancer IPs or whatever the arse else infra wise -- AWS will do all that right away for you for sure, while working out how to get a bunch of tin to do it will take longer, but it's like getting into first gear slightly faster than the guy beside you on a F1 race -- in the long run, this isn't where the time is going to be spent.
Lets say you're the ops dude instead, how do you move a few hundred hand built centos and java boxes to some kind of api based provisioning thingy with no config management? Yeah well.. There are a lot of options, if you're the one building it out, tbh the best choice for the infra part of the provisioning is just to pay someone to make this their problem. Redhat. Vmware. Whoever does Proxmox. Whatever. You'll lose E-Penis points but who cares your job is to help the business work faster. Even if you have do it yourself, there is a large ecosystem around things such as openstack or proxmox or smartos or whatever cobbler and so on, and this stuff is pretty friendly, it won't take you THAT long. Hire a contractor for a bit. Blammo.
The post talks about doing 'compute' then 'storage' and 'network' then 'stateless monoliths' and so on. Whatever. That's not reality. Why is the business paying for this? They want to be able to iterate quicker! CD is a tough goal with a legacy codebase, but they got used to agile and hell if it's not going to get faster -- the job is to enable that in any way possible, the tricky part of that is not going to be how you deal with infra, it's how you get the code through a pipeline and deploying painlessy.
Infra is a pain point. Legacy environments are fragile as hell and good operations folks are few and far between, and most of them are undervalued and a bit weird. -- but you personally, oarsinsync, are going to better off on prem for a few simple reasons in that world:
1) Total Ownership: Something goes titsup? Yeah, your team owns the entire show up until your ISP. Network borks? Storage slows down? It's all yours. Dig in. AWS develops cross-az latency? Log a ticket and 2 days later when they respond you'll have already forgotten it happened. Explain that to the business.
2) Perf: Don't believe the rubes; you'll be quicker on your own gear if only because you can move your noisy VMs to sep machines, set various limits/quotes/throttles that you can't on a public cloud, and you're not sharing spindles with 300 mongo vms.
3) Expierence: It's not exactly a business goal, but you'll learn more when you're having to stace some jvm at 3:00am and crying into a toilet than if you're just bouncing instances and getting away with shit code. If you just outsource all your problems forever you'll never be able to have an opinion on them, even if that opinion is just "I never want to use RAC again" or "mongodb is death" or whatever..
How exactly would a "private" cloud save you from these clearly cultural problems?
Going back on prem really helped, even if maybe it was just the "group project" to do it. Afterwards though I think there's a bit more buy in simply because the company has total ownership of the stack. Some team starts thrashing resources there's a bit more knowledge sharing and the stakes are higher (at least, the resources aren't limitless).
You're running your service in AWS and another team consumes it, then RDS starts getting slower what can you do? Blame AWS when you get a nasty call from a PM and go to lunch...
On prem? You're probably not far from the DBA who can explain what's happening and you can grab the consumer of your service to help too. It seems to just open thing up a lot..
YMMV.
I've seen a definite improvement culturally though.
We are tasked with moving to the cloud, and this is where everything stops. Over the years, instead of fixing the monolithic database design, the company instead kept beefing up the hardware for the database. A bigger cpu and more ram is cheaper up front than re-engineering the entire company infrastructure.
And now we are paying for it. Our db hardware now is the highest end hardware the db software supports. None of the cloud providers db support can handle it. We are forced to fix the monolithic design if we are to move to the cloud. It will be a multi-year effort.
Start from the top instead of the data. Moving various bits of the API to new implementations/microservices and modernize the stack -- new features, make 'em all standalone services and try to deal with your logic in the app instead of in the db. Move /send/giant_dildo_to?uzer=cyberpunk out of the monolith and to a sep service, and eventually you might get to the point where not a huge amount of the legacy code is actually being hit by anything besides your new code. At that point, the cost of redoing the persistence bit is way less than it was when you started.
Depending on your app, you might even get away with creating a new DB and using the orig as a kind of 'feeder' which you run on batch -- all options are open right? But obv it depends what you're doing.
And the company is willing to go through it because the public cloud offers so much?
Just curious if this was going to stop the overall project or not.
"[...] on-premise infrastructure on the other-hand brews a culture of mistrust, centralization, dependency and control. [...] These difference between on-premise data centers and public clouds influence how teams think, plan and execute. These are nothing but attributes of culture."
I'm currently working with a large financial services organisation that is moving to the cloud. It's incredibly painful, because of all the challenges involved in guaranteeing the safety and security of private data etc., but once these hurdles are taken once, it's incredibly liberating. The move is not just a technical one though, it requires a change of culture as well.
Obviously, this only makes sense in a world where individuals as well as corps need to present themselves online as a matter of course, but it might enable eg personal ID or data privacy tools that address many of the problems people have 'donating' their private data to commercial orgs.
From exprience, most organisations manage the privacy of their customers much much worse on their own hardware than they would if they used a public cloud provider.
Want to know how "privacy" really looks like? Take a look here - http://www.informationisbeautiful.net/visualizations/worlds-...
In the public cloud, as an engineer/employee/government I can just login to any given XEN/KVM/SmartOS/Whatever host that's running your instances and simply read your secrets and even if your opsec is perfect there's nothing you can do about it.
'privacy' in the cloud is not about mismanagement/good practice -- it's about who is able to act on it.
There is no way that anything you have running on infra you don't completely own and control will ever be private; to even a casual observer it's obvious that there is absolutely no defense or privacy possible in such an environment.
There are also a lot of companies who don't like storing data in a foreign country. What if the government was taken over by a Mr Trump who doesn't give about past agreements?
And using a managed cloud doesn't mean no data breach. You can still write SQL injectable websites that read data from the 'safe' cloud.
Is your point that everyone should move to the public cloud because you think it's "difficult" to build a "private" one?
Sorry to be blunt, but are you joking, an idiot or just completely unaware of the reality our industry outside of startup land?
Forgetting the compliance reasons, or that I have zero confidence that a combination of giant American corporations have any interest in protecting my data and that of my users, forgetting also that the performance differences are incredibly significant (take 8 15k sas's in raid 10 vs.. what? EBS with 2k 'provisioned IOPS' on a shared san over iscsi?), the cost savings are much more significant than you claim here.
I recently bought a half rack load of dell boxes which have like 380GB of ram each and 52 threads, and they cost around £4k each.
Even with a whole army of overpriced ops dudes to install openstack or kubernetes on all that, the cost we spent would be spanked in a month on AWS for similar capacity, and mine isn't a recurring one.
Why does it have to be so black and white for you folks?
A "private cloud" is absolutely the correct choice for a lot of people. If I were you I'd quietly delete that post, because I've absolutely judged you -- and probably unfairly -- on the content. On that, I wouldn't hire you.
Listen, both these things have a place, but they're NOT competing in the minds of those of us running serious infra. AWS allows us to prototype apps rapidly, to put project work on a credit card associated with a specific project budget instead of spending the shared budget, and for sure there are some nice services which do a bit of heavy lifting for us which helps speed that up.
After that phase though, very, very few people things actually need or make sense to stay there once it becomes a real product. Your scaling concerns are not being well served by 1000s of ec2 instances. Moving back in house is usually the smarter option.
If you're pumping loads of data into the internet, then you should be behind a CDN and the origin shouldn't be smashed to the point where you're dependent on throwing compute into the void to cope. Akamai or CloudFlare are cheaper than deploying 100s of ec2's because some shitty code can't handle your traffic.
If data is coming in, read a book on queues; you'd be pretty surprised what you can do with even two 1U boxes and a good software architecture. We could handle 100k+/sec of messages back in the java 1.5 and activemq days with 1U sparc boxes; how much do you recon it costs to do that today in AWS?
tl;dr - urgh. lets' talk in 10 years.
</rant>
might as well call it the inversion of scaling principle
Based on your reply, I suspect most people wouldn't want to be hired by you, either...
I share the sentiment that amazon & co are way too expensive beyond a simple proof of concept for example, even though for widely varying workload you may still need a public cloud.
I'm not attacking the person; calling someone an idiot does appear to be a personal attack, but it was asked as a question -- I was trying to convey my bewilderment at how anyone could share his views ("you like pepsi over coke?! what are you some kind of hippy?! -- etc!)
Come on, we're grown up and this is our area of expertise, why should we censor ourselves -- I'm not sure how else I could have expressed my view on the whole thing so succinctly tbh. You knew what I meant right off didn't you?
Are you offended on the authors behalf? Do you think that person, who is clearly an experienced technologist and who appears to have spoken at a conference, who has views on what is a pretty insider issue and has spawned a debate on the whole thing on hackernews is going to be offended because someone on the internet thinks it's stupid?
If he is; then I'd like him to know that it wasn't my intention to offend but also -- this is the internet and we all benefit from frank honest discussions, which I'd like to have and if you have issues with people questioning what you've written a bit aggressively then perhaps you shouldn't have written it.
It's not like I'm going full youtube or /b/ here... This is a forum, and no personal attacks happened.
It's a free world, my overreacting and being a grumpy bastard on a HN thread probably isn't enough to make that call tho.
I would start by addressing that concern, but it may come down to a matter of beliefs; i.e., you trust that 'the cloud' will be secure this next iteration (or the next, or the next), and they remain sceptical. Can't blame them.
I think the amount of enterprise DC's / colocation facilities is pretty stable but that almost all of the growth is in public cloud. Don't have any data to back this up though.
Public clouds often use colo facilities, especially for new/small regions. So colo != private.
For example MSFT investing $15B in its datacenter infrastructure. https://azure.microsoft.com/en-us/overview/datacenters/.
GOOG naturally don't need to colo anything, they have their own https://www.google.com/about/datacenters/inside/.
This article shows an analysis of DC vs. Public/Private Cloud IT Infrastructure: http://www.datacenterknowledge.com/archives/2016/07/06/cloud...
https://d262ilb51hltx0.cloudfront.net/max/1546/1*6rkj6frEx5B...