Hacker News Reader

Comment by theirishgoodbuy | Hacker News Reader

The way I look at it is: there are more tools in the toolbox than ever before. Which makes our judgement (the thing they really pay us for) even more important. Kubernetes, for example, is a specific solution to a specific problem. The solution is complex but so is the problem. If k8s give you the right trade-offs for your situation, then it’s not busy work.

Of course, there are plenty of project where judgement Is thrown out the window in favor of adding a buzzword to everyone’s resume. I’ve heard it called “promotion-based architecture”, as in you pick the technology most likely to get you promoted. (If that works, it says all sorts of not great things about your organization).

Regardless, I don’t think the availability of tools is the root problem. It’s and industry-wide lack of emphasis on identifying and understanding the problem first.

View on HN

Promotion based architecture is self fulfilling prophecy at least in BI/ data world.

I see everybody around me moving to cloud, without really good explanation why. Only reasonable thing I can see as an pattern is that cloud experience on top of data things gets paid 30% more. It made me consider cloud a lot.

I was considering switching to cloud, just so I can put in my CV "experience with migration to cloud".

For next person commenting that it makes sense: it doesn't with 200gb database and super predictable workload, growth and usage.

There are non-IT perspectives on this as well.

My company offers two deployment scenarios: host it yourself and cloud hosted (SaaS). Many of our customers choose the latter because their internal IT systems require more process, and they just want to get something up and running.

Just to add, at my last company it could take up to 6 months to get a VM provisioned on prem. We could provision what we needed in Azure on demand.

>For next person commenting that it makes sense: it doesn't with 200gb database and super predictable workload, growth and usage.

Depends on the company. I've been working for marketing agencies for the last 15 years and they're generally staffed by, at most, 1 IT person who is in charge of a third-party vendor relationship that offers managed IT services. Those IT resources (internal or vendor) don't specialize in data and often don't know how to deal with it well, predictable workloads or not, and often offer up solutions which are not appropriate (cost or otherwise) whereas there are managed solutions from cloud vendors that do. BigQuery, for example, handles compute and storage for you and rolls it into one reasonable query price. No need to worry about any management of anything there and just sit Data Studio (included) or any other BI tool (Tableau, etc) on top and you're good to go.

I get your skepticism and welcome it, but you're being a little rough. We're cloud native at my current company (I did a partial cloud migration at my last which was completed after I left) and it makes my life leading a Data Engineering and Data Science team MUCH easier without upfront hardware/software costs or long-term contracts, which were STAGGERING and left us with much more difficult to maintain, upgrade, etc hardware that took up most of my job as opposed to almost none of it today.

YMMV.

Partially depends but company with 1 IT person that is not data focused, there is low probability that they will have that amount of data.

For anything above what we have I think it weirdly depends on country/ salaries.

In US, 100k per year is no brainer for cloud as even 1 FTE would cost much more.

In non-western Europe, 100k is deal killer as you can hire 2 senior DBAs and you still have enough money for quite a reasonable server.

With extensive experience in this vertical, (marketing agency and analytics), 200GB is a _partial_ day for one datasource, let alone table or total DB size.

but does team have 1 IT/ data person?

Edit: sorry I see that you are the same person I was replying to. Then to understand it better, who is consumer of so vast amount of data?

The company has 1 IT person and uses a third-party vendor for IT support. We have several data engineers and scientists, but we are not specialists as DBA or cloud infrastructure; it's just one very small part of our work.

Interesting, thats correct, its segment of the companies that I didn't take into consideration. Thanks

> 1 IT person who is in charge of a third-party vendor relationship that offers managed IT services

That is something completely different from migrating your inhouse applications to the cloud by introducing kubernetes. I think your example show nicely that there is not one solution for everything. As an industry we have to put way more emphasis on deciding tradeoffs by understanding requirements and risks.

I see everybody around me moving to cloud, without really good explanation why.

People just buy into the "cloud" marketing. They don't have the ability to think and reason, and so don't understand that "cloud" just means "renting someone else's computer."

I built a complex in-house medical system. Quick, reliable, and liked by the users.

I was in the middle of adding a major new feature when all of the management in the IT department quit. The new people immediately decided that the whole thing had to be "in the cloud." I was removed from the project, and they hired three people full-time to rebuild it.

That was three years ago. The new system is still not online. The users are still using my old system, and the feature I was working on never got added because my presence and input was not welcome because I "don't understand the cloud." So I got moved to other projects.

People talk about "the cloud" with the same fervor and language as members of a cult. And you will be an outcast if you dare challenge their way of thinking.

Given what you explained, it seems to me that the cloud was an excuse given by the new overlords to just get you out off the way. Cloud is fashionable now, but any excuse would work for them. This seems to be a power grab.

I don't think it's about power. Because of the way the organization is structured, I am isolated from the IT department.

Moving me out brings no advantages to either part of the org. In fact, it's probably a disadvantage to IT because it has to burn three people from its allocated headcount. Where previously, it had a bonus person that didn't come out of the IT budget.

It's always about power and politics when bone-headed decisions like that are made. You correctly identified that _you_ did not report to IT, therefore _you_ were a threat to IT's power. Maybe you could have been re-orged under IT, but they probably saw you as a troublemaker who wouldn't kotow.

At the risk of sounding cultish: I would argue “the cloud” is a bit more than just renting someone else’s computer. Hosting companies existed long before the “cloud”.

The difference is the abstraction layer. I would define “the cloud” as a layer that abstracts away the physical infrastructure. (Which works until it doesn’t but that’s a longer comment.)

You can even apply “the cloud” to your very own fleet of computers with the right software.

Like I said in the original comment, it’s a specific solution to a specific problem. Whether it’s right for your system I have no idea.

You can even apply “the cloud” to your very own fleet of computers with the right software.

If it's on-prem, it's not a cloud, it's a fog.

At least, that's what I've been calling some of my deployments.

> People just buy into the "cloud" marketing. They don't have the ability to think and reason, and so don't understand that "cloud" just means "renting someone else's computer."

It sounds you're trying to dismiss and downplay solutions to problems you're oblivious to.

Even if you want to simplisticly frame "the cloud" as "renting someone else's computer", keep in mind that:

* It's not one computer but as many as you'd like, and get them at the click of a button,

* These are computers which are managed 24/7 by a team of highly trained specialists,

* These computers can be located anywhere in the world and simultaneously in multiple regions,

* These computers are designed to be fault-tolerant and handle (and survive) way more stuff that can conceivably be thrown at them.

> People talk about "the cloud" with the same fervor and language as members of a cult.

I'm sorry to say but you sound like you're militantly opposed to a solution to problems you either don't understand or refuse to understand.

"The cloud", even when following the simplistic and clueless belief that it's just "renting someone else's computer", solves whole classes of technical and business problema that your box under your desk cannot solve.

> Only reasonable thing I can see as an pattern is that cloud experience on top of data things gets paid 30% more.

You say that like a pay rise of 30% is not a good enough reason all by itself for many people.

> For next person commenting that it makes sense: it doesn't with 200gb database and super predictable workload, growth and usage.

Perhaps not for technical or business financial reasons, but those are not the only possible reasons someone might do something. As mentioned before it can make a lot of sense to migrate to cloud if it means you can get a 30% pay rise.

There's a finance angle behind cloud stuff too that's irrestible for the bean counters: cloud stuff is operations expense, on-prem is a capital expense. Unfortunately these folks are heavily incentivized to favor OpEx

I'm not a bean counter, all I know is those guys at my last job would rattle off about it like zombies. IIRC its a tax thing

Opex is fully deductible each year the expense occurs, wheras capex requires amortized depreciation over the life of the purchased item. Makes it harder to calculate taxes.

Should we call it Amortisation as a Service (AaaS) then?

Is that really the only benefit?

I mean, I'm not a bean counter. I'm just guessing that's why they like it.

Wrong. People has been leasing on-prem hardware for decades before the cloud existed.

Depending on taxation rules you might want to buy or lease.

At this employer it seems most of the stuff in their datacenter was purchased.

It's not a tax thing, long-term leases or colocation is a liability for acquisitions. (You could also build a revenue-positive company, but that's the loser way out, apparently.)

It makes sense when you aren't a datacenter hosting company as your core business. Once you're done with your DRP and BCP it's highly unlikely that your server-under-the-desk (as it usually is) is worth the risk.

By the way, moving 'to the cloud' never means a specific thing because people have made words (intentionally?) vague to the point where you have to explicitly specify all the factors you take into account with your work in order to figure out which 'cloud' they had in mind.

Running a static workload doesn't require elasticity, but a 'cloud' isn't just elasticity. If you want "a program on a server with some storage that is always there" without having to deal with hardware, software, networking, storage, backup, maintenance, upgrades, service contracts etc. then an arbitrary virtual machine where someone else takes care of the rest makes total sense.

And its easy to compare the cost of the cloud running your hardware to the cost of physical hardware. However, its much more difficult to compare the indirect costs between the two, and that's where I think many people go sideways.

Hell, I'm working with ~200TB and it still doesn't make sense if you work out the costs.

As far as I can tell it only makes sense if you have a ton of data, and only ever use a fraction of it at a time, for one-off jobs, infrequently.

It's a bit dismissive to just focus on scaling as a cloud advantage. You also have lower maintenance, easier redundancy, lower chance of outages, easier backup and more such non-functionals. A 200GB DB costs about $50/month on AWS (other providers are less), that's not an interesting amount of money for most businesses.

Thats way off. With 64gb ram we have around $50 month on hetzner yes. But from asking around the other BU with very similar size and usage to ours, somehow they are paying around $2600-5000/ month on azure

Are they using Oracle or MSSQL where you're paying insane software licensing costs?

With Postgres[1], you would need to be using an absolute beast of a server to be paying that much, even if you're paying the 2x premium to go month-to-month, and a low-powered machine could be around $50/month for a 200GB DB.

[1] https://azure.microsoft.com/en-us/pricing/details/postgresql...

Yeah, its true that they are using MSSQL.

Based on link you provided, I calculated that 40gb memory would be $290 and 80gb $570 per month (in UK).

So price seems still quite steep, considering that same specs on-site would be ~$2000 one off. (8 months return on investment).

Then for this server, I honestly don't believe its easier to do maintenance in cloud, as you need to do lots of initial configuration there (be it IAM, permissions, firewall etc. ) compared to have it just in local network

> With 64gb ram we have around $50 month on hetzner yes.

A single box on Hetzner is by far not comparable to, say, using a managed service as AWS RDS. It's not even apples to oranges, and more of apples to freshly-squeezed orange juice served by a buttler. Think about it: do you get any form of fault tolerance with your single box?

Thats why I am saying that $50/ month is way off. The price for that would be much higher

> Thats why I am saying that $50/ month is way off. The price for that would be much higher

Ah yes I agree, sorry for not being clearer. I wanted to add on to what you said.

Even though I'm a big fan of Hetzner and a happy customer for years, it's important to know what we're buying and be mindful of what we're not having as a tradeoff.

No one has given you a good explanation for why to move to the cloud?

Let me fix that for you! :)

I work for a Fortune 200 company who has their own data centers and has for decades. We just completed the construction of two new data centers about four years ago at a cost of $110 million dollars. Those new data centers are now at 70% of their capacity for power requirements. You should take a look at the specs on Intel's latest server-class processors. It's insane! We're talking half horse power and higher for each processor - and you want blades full of these things and then a rack of those blades and a row of those racks! Data centers now consume more power than industrial manufacturing! Most maddening of all - most of that power is going to go to heat, and guess what? You need giant chiller units to cool the place down. The power costs alone are insane.

You also lose agility. Need a new server? Go to your favorite cloud console and provision it. Better yet, use a serverless architecture and don't even worry about servers! Your own data center? Right now there's a 4-6 month wait time for new servers and storage equipment, and an 8-12 month wait time for networking equipment. Not exactly agile, is it?

You already know you need to staff to manage and patch your servers but you're also going to need staff to procure and manage your software licenses - and those licenses aren't cheap! Don't forget the ongoing tail - you pay for the license and then you get to pay 20% per annum forever after for that license. Those licenses also restrict your agility - you're pressured to use the software that's licensed in order to get better value. Try working with a procurement-based architecture!

You think you're going to avoid the software licensing hell by using open source? Good luck with that! That means you are responsible for the packaging, distribution, and support of whatever it is you're utilizing. That's more staff you need. They will inevitably miss patching some critical vulnerability that's going to land you in the news!

I can go on, and on, and on. But here's one thing I can say about applications that we've moved to the cloud: they cost 30% as much to host in the cloud as they do on-premise, and that's not even accounting for the entirety of all the costs I enumerated above! We have ultimate flexibility and agility, and we can quickly utilize managed open source platforms to solve business problems. On-prem? You lose all of that.

Faster, cheaper, better - that's why people are moving to the cloud.

Open source clearly does not mean packaging your own software. This casts a shadow of doubt on the rest of your valid points.

It does if you're not paying for it. There are plenty of companies that will gladly handle the packaging, assure all the interdependencies work well together, and even support the effort. They also do release management, ensuring that everything is on the most recent version that all works together. That has a cost. You can either pay someone to handle that work for you (smart - you and many other companies are sharing that cost) or you can do that work yourself (not so smart - now you shoulder all the costs).

Just as an example, consider Hadoop and its ecosystem. Another example is Elasticsearch. There's work involved in making these platforms production-ready and keeping them and their dependencies up-to-date and ensuring all your applications in your portfolio are using the same set of software. Most organizations are not prepared to take that on. So they turn to a vendor who will do it for them for much cheaper than they could do it themselves.

That's why the saying is "free as in freedom, not free as in beer" because open source software is not free as in beer. At least not when factoring in the total costs.

> Open source clearly does not mean packaging your own software.

The packaging argument was made in the context of complying with software licenses. Responsible companies which perform due diligence on the software they run have to track the provenance of all software that ships as part of their dependency closure. If you want to ensure you're not vulnerable to lawsuits then you can't simply apt-get stuff from a PPA. You need to build it yourself, and track exactly what goes into that build.

Meanwhile, if you opt to run managed service from a cloud provider, you don't have to bother with that because that's not your problem (or liability) anymore.

It is your problem though, your customers are affected.

Do people successfully lawsuit their cloud providers for downtime?

But that should apply to all readily available container images on docker hub as well, right? Conceptually that is not different from some guys PPA.

> But that should apply to all readily available container images on docker hub as well, right?

Yes, and it does indeed apply to all readily available container images.

In fact, it applies to any and all software packages put together by third-parties.

I mean, who in their right mind downloads random stuff from the internet and expects to just drop it in production software which you build your business upon?

Go to your favorite cloud console and provision it.Better yet, use a serverless architecture and don't even worry about servers! Your own data center? Right now there's a 4-6 month wait time for new servers and storage equipment, and an 8-12 month wait time for networking equipment. Not exactly agile, is it?

Sure, that all works, as long as you have infinite money. My last job was at a company that was heavily invested into a serverless architecture, using the full suite of AWS tooling. They did an audit of one of their codebases and they found that generating a particular piece of data cost $20, almost entirely in fees to AWS. The company's roadmap involved scaling up the generation of this data, so they embarked on a huge process of optimizing and refactoring, to try to get the cost down. This effort tied up probably 40-45 engineers for at least six months. Probably cost them roughly 3 million dollars in salary alone, let alone opportunity cost in terms of other projects not delivered. No one amongst the management ever seriously considered moving to an on-prem solution, even though it was blatantly clear that the only value the company's massive usage of AWS Lambda, SQS, RDS and other managed services was delivering was additional profit margin for Amazon's balance sheet, and resume line items for the engineers who now got to check off the "used Terraform" box when applying to their next job.

You must have missed the part where I said But here's one thing I can say about applications that we've moved to the cloud: they cost 30% as much to host in the cloud as they do on-premise, and that's not even accounting for the entirety of all the costs I enumerated above!

Every app I've migrated to the cloud so far has resulted in a 70% cost reduction, but I'm also not trying to take my entire application portfolio to the cloud. To wit, we're not planning on abandoning our data centers anytime soon and that's not just because we can't move our applications to the cloud fast enough. At this point in time we recognize there are some applications best kept on-premise - but they're a minority, maybe 20% tops of my application portfolio. That means 80%+ of my application portfolio can be moved to the cloud, makes sense to move to the cloud, and I can save 70% on costs by doing so. That's a fantastic deal!

Have you done an audit in the other direction? How many "cloud native" apps do you have, which would realize similar cost reductions from moving to on-prem? The point I was trying to make is that many companies, especially newer startups, don't even discuss on-prem as an option. They build everything cloud-native from day 1, which then leads to them painting themselves into a corner like I described above. They're having to totally redesign and rearchitect their application, not because their business requirements dictate it, but to conform with AWS' limitations.

It's May 6, 2022. You're launching your startup next month. Are you going to host on-prem? What, with server equipment backlogs now 4-6 months, and that's if you're a large enterprise customer? Network equipment backlogs are now in the 8-12 month range. Not to mention the cost of needing conditioned power, cooling, staff, space, etc. You're going to utilize the cloud. It'd be insane to do otherwise.

You do raise a fair point - what about the applications that are more economical to host on-prem? In my own portfolio I estimate 20% or so of my applications make more sense to host on-prem. What to do about those?

The most practical option is to mandate every application you build in the cloud have a plan for how to run on-prem. Note those that would be very difficult to run on-prem - those are your key cloud dependencies. For example, in my own portfolio I have a couple of applications depending on Lex and therefore would be very difficult to host on-prem.

Pay special attention to your so-called "crown jewels" - your apps that distinguish you from your competitors. If those were dependent on a key technology such as Lex and would be difficult to migrate on-prem then you need to identify other cloud providers you could migrate to. In this particular example I know Microsoft's and Google's clouds both offer similar services.

If you design and build your applications with no thought of the future then you're already in a bad position.

It's May 6, 2022. You're launching your startup next month. Are you going to host on-prem? What, with server equipment backlogs now 4-6 months, and that's if you're a large enterprise customer?

On-prem or cloud, why would I be waiting until 1 month before launch to provision anything?

You're pretending like on-prem is full of all these physical limitations, but Amazon, Microsoft, Google, Oracle, et. al. can just magically make hardware appear out of nowhere. It's the same hardware, subject to the same supply limitations, and I'm going to be paying either way. Either I pay more per hour for my cloud instances, or I pay a bunch up front to provision my on-prem servers.

Cloud makes sense if the load on your service is intermittent or unpredictable enough that you're okay paying AWS's premiums in order to have the ability to scale on demand. But if the load on the services is knowable, then hosting that service on the cloud is paying a 25% markup for no reason whatsoever. AWS is by far the most profitable portion of Amazon's business units. Those profits are coming from somewhere. They're coming from your business.

>>I see everybody around me moving to cloud, without really good explanation why.

In that case you certainly aren't looking very hard for that explanation.

The rationale of moving to "the cloud" is crisp and very well-defined, to the point where even AWS makes it their point to explain in very clear terms in their Architecting With AWS courses.

Basically it's all about being able to scale without bothering with procurement, not having to manage everything down to power bills and rentals, turning big capex into small opex, and being able to deploy globally and reliably with a click of a button.

That, however, comes at a steep cost. The more serverless you get, the higher the price tag.

Also, beyond a certain scale you're better off going back to managing your own infrastructure.

> Only reasonable thing I can see as an pattern is that cloud experience on top of data things gets paid 30% more. It made me consider cloud a lot.

People with cloud experience are paid more because they can deliver more value. A random guy with, say, experience in EC2 and CloudFormation and CloudWatch, can single-handedly put together a robust fault-tolerant webservice that's deployed globally and automatically scales to meet any demand fluctuation.

There are 10, probably more, cloud consultants per developer. Hard to argue against that.

There is nothing to say against a cloud version of your office suit but otherwise it is quite underwhelming. Short lived applications with a 90% API because the developer want to keep the door open to imprison the users once the application has captured a critical mass of users.

I host on AWS because I don't pay the fairly high invoices. Less domain and TLS maintenance for me. Personally I rent a server and host my personal stuff there including software repositories.

edit: Guess that technically would also qualify as cloud, but I think a differentiation is necessary. Cloud mostly means product x or y instead of just another hosted server.

Hired onto a project as a specialist. Funders had hired a project manager also. Project manager wanted to use technology X ( which has long since faded ) instead of well entrenched and dev-hireable technology Y because technology Y had not yet entered the trough of disillusionment and anyone who could honestly say they had managed a technology X project could 10x their hourly. Money was ignorant about this level of detail. PM won, as they do. Whole thing a money pyre.

I call it Resume Driven Development

Yes, this has been my observation as well. I also see this happening concurrently with what I call the "shiny object" problem. Developers have their eye on a shiny new library, framework, language, etc. and have a seated desire to use it at the expense of it being an inappropriate choice for the company.

I believe these two phenomenons are producing a positive feedback loop in the industry. Selected technologies address one challenge, but introduce complexity. The complexity becomes difficult to manage. So other technologies are incorporated to manage the complexity. In the midst of all this, core technologies are replaced, swapped out, or transitioned to under the desires of the dev(s). For example, switching from one js rendering library to another, while preserving the old legacy code. The complexity footprint keeps growing and it doesn't stop because the engineers themselves aren't entirely committed to the project. They can incorporate the new technology to the project, pad their resume, and bail to a new employer if things grow out of wack.

I worked with someone who did this. I'm glad he has moved on. He would not do basic tasks correctly. Had an attention span of a toddler. He was DBA and couldn't google simple problems. I asked him to do something he would say give me 20 minutes. 2 weeks later he would say "IT CANT BE DONE". Then, I would do it myself in a few minutes after googling the error.

Instead of applying knowledge, he would bring up all these buzzwords to meetings and not really understand what he was talking about. I get angry just thinking about it.

Hmm, incompetent folks exist and need to be let go. However, I think it is a slightly different category than RDD.

> I worked with someone who did this.

Nothing in your comment points to resume-driven development. You even failed to mention any project or design decision. You just decided to get angry at a coworker because of something.

"promotion-based architecture" aka "CV-driven architecture" :)

No, it's called "résumé-driven design".

I’d argue the quality of tools is the problem. I am starting to view every new tool as just a new set of log messages I have to sort through to figure out why things aren’t working.

I spend more time trawling GitHub issues to find workarounds than I do actually using the tools.

> It’s and industry-wide lack of emphasis on identifying and understanding the problem first.

Or just no real incentives of the people involved to do so. As a dev, I don't get any real credit for biz outcomes.

the auld mortgage driven development strategy

This was very well put. +1

> “promotion-based architecture”

That's a great way to describe the phenomenon.

Reply on news.ycombinator.com