Hacker News Reader

Comment by phpnode | Hacker News Reader

I really don't understand why the industry seems to have lost sight of this. It's really common to see super complicated, incredibly expensive, but highly scalable cloud deployments for problems that can be trivially solved with one or two dedicated servers. Even the mere suggestion of renting a dedicated server provokes scorn from devops teams. The overall difference in cost when taking into account all of the complexity, feature-lag and general ceremony must be at least 10x and maybe even closer to 100x. It's madness.

View on HN

The allure of our industry to mime FAANG leaders might be one of the reasons we're at this stage. Most products could be safely run on simpler setups.

I suspect that VendorOps and complex tools like kubernetes are favored by complexity merchants which have arisen in the past decade. It looks great on resume and gives tech leaders a false sense of achievement.

Meanwhile Stackoverflow, which is arguably more important than most startups, is chugging along on dedicated machines.

1: https://stackoverflow.blog/2016/02/17/stack-overflow-the-arc...

Software development and related things are known to have fantastic often free online resources for autodidacts.

It seems like the trend in this space is to jump directly to the highest layer of abstraction. Skipping the fundamentals and point to $buzzword tools, libraries and products.

You see glimpses of this in different forms. One is social media threads which ask stuff like “How much of X do I need to know to learn Y?” Where X is a fundamental, possibly very stable technology or field and Y is some tool du jour or straight up a product or service.

"complexity merchants" - I love this term. It aptly describes beneficiaries of earlier IT fads. Remember UML? Remember WS-*? One could come up with a long list of technologies that were a bonanza for billing consultant hours but no so great for getting to solutions.

I remember architecture astronauts.

The debacle that was J2EE is hard to top (but I'm sure some have!). I still have battle scars from OSGi.

CORBA belongs there. And perhaps even the semantic web stuff. Definitely XML!

To be fair - J2EE (and its associated technologies like SOAP, XML) is a bit like alcohol. It is great when used in moderation, but gives you headaches when used excessively.

XML is great - if you don't believe it, then you might be interested in the pains of parsing JSON: http://seriot.ch/projects/parsing_json.html And don't get me started on YAML...

SOAP, don't forget SOAP.

SOAP is great when it works. Ingest a WSDL file and boom you've got an API client.

And it almost never works if you use different frameworks.

Heh, lot of emphasis on "when it works".

The dedicated server deployments I worked on at smallish software companies 10+ years ago wound up being really annoying. I enjoy sysadmin type stuff and this idea has tempted me, but I think it’s a false economy in most cases.

The incremental cost of being on a cloud is totally worth it to me to have managed databases, automatic snapshots, hosted load balancers, plug and play block storage etc.

I want to worry about whether our product is good, not about middle of the night hardware failures, spinning up a new server and having it not work right because there was no incentive to use configuration management with a single box, having a single runaway task cause OOM or full disk and break everything instead of just that task’s VM, fear of restarting a machine that has been up 1000 days etc.

100%. There are definitely cases where managing your own dedicated server infra (not hardware, just the software part) makes sense.

For me, the allure of the cloud is not some FAANG scalability fad when its not required, but automatic OS patching, automatic load balancing, NAT services, centralized, analyzable logs, database services that I don't have to manage, out of the box rolling upgrades to services, managing crypto-keys and secrets, and of course, object storage. (These are all the things we use in our startup)

I'd go on a limb and say dedicated servers may be very viable/cost effective when we reach a certain scale, instead of the other way round. For the moment, the cloud expenses are worth it for my startup.

The key is to use some kind of config management, even for a single box. The problem you need to solve is basically to separate your configurations from the system default and make it easy to bring the system into a known good state.

I do think there's something in there about the immediacy of a replacement if a currently-live box falls over. Rough rule of thumb says it's hours for a physical box, minutes for a VM, seconds for a k8s container, a hopefully smaller number of seconds for a serverless cold start. Each of those has ergonomics different enough that it might matter even with config management that rhymes across the options.

We should design a protocol that bootstraps newly booted systems via some protocol.. We could call it the Bootstrap Protocol, or bootp for short...

Sure, but if you are considering a physical box then you have presumably already decided that you are OK with it taking longer to replace.

This is now eating into the cost savings over a cloud deployment.

How so? You should still be doing those things with cloud deployment anyway. The alternative is (the pretty terribly named) "click ops" where devops folks have to fiddle with some cloud UI settings to make production changes or go through wizards to attempt to replicate an environment.

If you are doing your environment config as code it ultimately shouldn't matter if your target is a dedicated server, a vm or some cloud specific setup configured via an api.

But most cloud deployments have config management too. Using Ansible/Terraform/Kubernetes on bare metal isn't much different than using Ansible/Terraform/Kubernetes in the cloud.

You don't escape configuration with the cloud unless you are talking about the most simple of use cases on a platform like Heroku.

Ansible

yes, or Puppet, Aviary, Shell, CFEngine, Salt, Capistrano, etc...

Doesn't really matter which one, the point is that you have one that can build you a new box or bring a misbehaving box back to a know good state by simply running a command.

Aaand that's why k8s seems like a good step in the right direction. It has all - okay, absolutely not all, but a good number of the important ones - of these concepts "built-in".

Sure, the actual pile of Go(o) that drives it can be improved (and it is indeed improving).

That said, the hard truth is that your approach is the correct one, almost all businesses (startups!) overbuild instead of focusing on providing value, identifying the actual niche, etc. (Yes, there's also the demand side of this, as the VC money inflated startups overbuild they want to depend on 3rd parties that can take the load, scale with them, SLAs and whatnot must be advertised.)

> k8s seems like a good step in the right direction

Kubernetes is fantastic! Whenever I see a potentially competing business start using Kubernetes, I am immediately relieved, as I know I don't have to worry about them anymore. They are about to disappear in a pile of unsolvable technical problems, trying to fix issues that can't be traced down in a tech stack of unbelievable complexity and working around limitations imposed by a system designed for businesses with traffic two-three orders of magnitude larger. Also, their COGS will be way higher than mine, which means they will need to price their product higher, unless they are just burning through VC money (in which case they are a flash in the pan).

Good stuff all around.

k8s is almost the polar opposite of this idea for most companies. At minimum, a half dozen daemons spread across multiple hosts for.. what? Making the easy things built-in and everything else much more complex than it needs to be?

And those daemons use constantly 25-30% CPU.https://github.com/k3s-io/k3s/issues/294

I think it depends on the distribution. Microshift for instance seems like a good embodying of the concept that the parent poster was referring to.

I don't agree, Ansible would be a much saner approach for a few dedicated boxes than k8s

(I haven't used it in a few years, when I tried to deploy OpenStack with it, so if this is outdated, please, correct me!)

Ansible is imperative, it can work toward a static state, and that's it. If it runs into a problem it throws up its SSH connection, and cries tons of python errors, and gives up forever. Even with its inventory it's far from declarative.

k8s is a bunch of control loops that try to make progress toward their declared state. Failure is not a problem, it'll retry. It'll constantly check its own state, it has a nice API to report it, and thanks to a lot of reified concepts it's harder to have strange clashes between deployed components. (Whereas integrating multiple playbooks with Ansible is ... not trivial.)

What about deploying prebuilt docker image on dedicated server?

This amazing playbook spins up a ton of dockers:

https://github.com/spantaleev/matrix-docker-ansible-deploy

What about it? It can be easily done with Ansible. I managed a distributed infra with containers orchestrated by Ansible, worked fine.

it would be much less moving/fragile parts in prod. I am not an expert in this topic, and trying to learn about different approaches.

And then you have to deal with the complexities of K8s…

Yes, ... and? Those are much more "standardized" than whatever else any team cooks up. (And k8s along with Go is steadily improving, so I don't see this as "let's use WordPress because its the platform that has the most answers on StackOverflow".)

And even if k8s puts on too many legacy-ness, there are upcoming slimmer manifestations of the core ideas. (Eg. https://github.com/aurae-runtime/aurae )

It doesn’t matter if it adds unneeded complexity as long as it is standardized?

And now you’re suggesting that people use a “simpler” non “standard” implementation?

You need to solve a bunch of SRE/DevOps problem, fearless deploys, rollbacks, canary rollout, blue/green deployment, HA/LB, backups, dev environments, supporting multiple teams, yadda-yadda.

So of course you can implement a minimal complexity solution or you can use something "off the shelf".

k8s is complexity and some of it is definitely unneeded for your situation, but it's also rather general, flexible, well supported, etc.

> you’re suggesting that people use a “simpler” non “standard” implementation?

What I suggested is that if k8s the project/software gets too big to fail, then folks can switch to an alternative. Luckily the k8s concepts and API is open source, so they can be implemented in other projects, and I gave such an example to illustrate that picking k8s is not an Oracle-like vendor lock-in.

I think there's a lot of room to bring the best of both worlds together. Some are moving on this already, for example Hetzner Cloud load balancers can also have dedicated servers as targets.

If you've got your configuration properly automated, nothing prevent you from redeploying your bare-metal nodes every weeks. The opposite can actually also be true, it may become risky at some point to reboot a VM with a high uptime.

I'm completely clueless about any server administration beyond a LAMP stack on a single machine serving a few dozen people

I always hear about all the great stuff you get practically for free from cloud providers. Is that stuff actually that easy to set up and use? Any time I tried to set up my LAMP stack on a cloud service it was such a confusing and frightening process that I ended up giving up. I'm wondering if I just need to push a little harder and I'll get to Cloud Heaven

It's a mixed bag. It's more standardised, so once you know how to set up one app, you can mostly just repeat it multiple times as needed. Even for custom things, I can't remember the last time I started a CloudFormation stack from scratch - it's mostly copy&customise.

Being able to say "I want a clustered MySQL here with this kind of specs" is much better (time-wise) than doing it on your own. The updates are also nicely wrapped up in the system, so I'm just saying "apply the latest update" rather than manually ensuring failovers/restarts happen in the right order.

So you pay to simplify things, but the gain from that simplification only kicks in with larger projects. If you have something in a single server that you can restart while your users get an error page, it may not be worth it.

The big problem for server operations is not the damned servers but all of the infrastructure you need to keep them running, like power cooling network/storage and internet connectivity.

The cloud is not easy but damn trying to get cooling and power efficiency of an small server room anywhere near the efficiency levels most big data-center publish is next to impossible as is multi vendor internet connectivity.

With the cloud all of that kind of goes away as it's managed by whatever data-center operator that cloud is running on but what people forget is that that is also true for old fashioned colocation services which is often offering a better cost/value then cloud.

And while it's definitely harder to manage stuff like AWS or Azure because it bleeds a lot of abstractions small scall vpc providers hide from you or that you dont really get with a single home server, it's not hard on the scale of having to run a couple of racks worth of vmware servers with SAN based storage.

I think the problem is configuration. Every thing that has its own configuration is basically a new language you must be able to speak perfectly. Configuration languages are not statically typed so when you make an error it typically does not tell you you made a mistake, and where, and thy it is a mistake.

With Cloud stuff you have more configuration to do because it is about configuring virtual servers etc. Instead of carrying the PC in a box to your room you must "configure it" to make it available.

Laravel Forge + Digital Ocean (or Hetzner, your own machine, Vultr ...) is a pretty good solution. You can up & downgrade as you see fit or spin up a new server in a few min (any specs you need, from scratch or using an image of one of your existing machines, it installs what you need and you can of course add whatever is still missing).

DO databases have backups you can configure to your liking, store them on DO Spaces (like S3). DB user management is easy. There's also cache servers for Redis.

You can add a load balancer and connect it to your various web servers.

I think it took me about 30 min to setup 2x web servers, a DB server, cache server, load balancer, a storage server and connect them all as needed using a few simple forms. Can't really beat that.

By your own estimate -- how confident do you feel that these servers and services are secure? "Setting up" web servers to perform their function is rather easy, in my experience. Ensuring those servers can withstand standard-issue hacking attempts, not so much, especially within just 30 minutes.

The added value of these types of service, I think, is that they're fairly well set up with their provisioning script. You're paying for this service after all and if it appears they can be easily compromised "by default" then ... there would be many problems.

If you have any more info or opinions then please do share.

I'm no expert on Linux security but what I've seen when provisioning a VPS from anybody is a system with the base OS installed and that's it. SSH on port 22. Root account active. Accounts not needing a private key to login. No firewall. Known vulnerabilities in base packages unpatched.

Lots of articles around the internet about hardening a Linux server, the ones I've tried take a bit more than 30 min to follow the steps, a lot longer if I'm trying to actually learn and understand what each thing is doing, why it's important, what the underlying vulnerability is, and how I might need to customize some settings for my particular use case.

As someone who also runs a Laravel site on DO (but doesn't use Forge so can't comment on whether it handles any additional security configurations) buying a cheap DO droplet is very much handling your own infrastructure, it comes with reasonable enough defaults for most people out of the box but if you're expecting any decent amount of traffic it definitely needs to be hardened a little more. Also other administrative tasks like adding swap space if you're like me and trying to get as much performance as possible from as little spend as possible.

a bash script that auto installs everything will do. It has to be maintained though. You also lock down and harden the server. Running a stable OS helps.

I've run a few debian servers for years with little issues. Set up auto-update (very easy) for security updates. There's really little to go wrong.

I'm sure you can find example setup scripts online (configure autoupdates, firewall, applications, etc.), should be a matter of running 'curl $URL' and then 'chmod +x $FILE' and 'bash $FILE'. I didn't need configuration management (I do use my provider's backup service which is important I guess).

Something like this: https://raw.githubusercontent.com/potts99/Linux-Post-Install...

(seen in https://www.reddit.com/r/selfhosted/comments/f18xi2/ubuntu_d... )

Totally, though I think in a fast paced/under-resourced environment incentives matter, and ephemeral cloud vms incentive admins to use configuration management, whereas it's easy to take a shortcut in the moment and quickly manually install something/edit config on a rarely-changing dedicated host.

Obviously the same can be said for long running VMs, and this can be solved by having a disciplined team, but I think it's generally more likely in an environment with a single long running dedicated machine.

This here is the trick. I like to do blue/green deployments. This workflow forces you to script (and implicitly test ) the full server+application deployment. With everything scripted, it just becomes a matter of deciding to rebuild a new machine every week/month/n-deploys which makes you comfortable. Plus, you get a hot spare for free.

> The incremental cost of being on a cloud is totally worth it to me to have managed databases, automatic snapshots, hosted load balancers, plug and play block storage etc.

Hetzner has all of this except managed databases.

On their managed servers you can install data bases via KonsoleH and they manage it. (Never used the managed servers myself. MySQL and PostgreSQL?)

https://www.hetzner.com/managed-server

The webhosting packages also include 1..unlimited DBs (MySQL and PostgreSQL)

https://www.hetzner.com/webhosting/

> install data bases

does it have automatic backup and fail over?

Looks like it has backups, to an extent [1]:

> With booked daily backup or the backup included in the type of server, all data is backed up daily and retained for a maximum of 14 days. Recovery of backups (Restore) is possible via the konsoleH administration interface.

But i get the impression that the databases on managed servers are intended for use by apps running on that server, so there isn't really a concept of failover.

https://www.hetzner.com/legal/managed-server/

which is most complicated part.

EC2 or on-premise you need monitoring to detect issues before they become issues.

A single drive on a single server failing should never cause a production outage.

A lot of configuration issues can be tracked down to self-contained deployments. Does that font file or JRE really need to be installed on the whole server or can you bundle it into your deployment.

Our deployments use on-premise and EC2 targets. The deployment script isn't different, only the IP for the host is.

Now, I will say if I can use S3 for something I 100% will. There is not an on-premise alternative for it with the same feature set.

Fine, but RAID-Z2 in a dedicated server is good enough or overkill for most people and it's free.

HackerNews went down because two SSDs in a mirror went down at the same time. This was due to a hours on SSD bug. While Raid-Z2 works, and provides great reliability, two-servers replicated, both with raid-z2 and a nightly offsite backup and as long as you are testing your backups it's a solid plan.

And AWS goes down about once a year because US-East-1 has some weird failure. Your point?

The point of DevOps is "Cattle, not pets." Put a bullet in your server once a week to find your failure points.

What about simply running k8s on metal? You wouldn’t be able to auto-scale (unless you automated buying and selling the hardware provision from the DS) but it would be nice.

Then you have to manage k8s, and based on the number of companies I've helped move away from that model (to PaaS mostly) it's no fun either.

What PaaS providers / platforms do you like? I want to move some personal shell scripted boxes to PaaS, but dropped the idea with the decline of Heroku. Tried Dokku and struggled, but it seems like I should give it another try.

Maintainer of Dokku here:

I'd love if you jumped into our Discord/Slack and brought up some of the issues you were seeing so we can at least make the experience better for others using Dokku. Feel free to hit me up there (my nick is `savant`).

Thank you for the kind words and I'll check out your Discord when I next take a pass at it.

Let me preface and say that I'm an application dev with only a working knowledge of Docker. I'm not super skilled at infra and the application I struggled with has peculiar deployment parameters: It's a Python app that at build-time parses several gigs of static HTML crawls to populate a Postgres database that's static after being built. A Flask web app then serves against that database. The HTML parsing evolves fast and so the populated DB data should be bundled as part of the application image(s).

IIRC, I struggled with structuring Dockerfiles when the DB wasn't persistent but instead just another transient part of the app, but it seemed surmountable. The bigger issue seemed to be how to avoid pulling gigs of rarely changed data from S3 for each build when ideally it'd be cached, especially in a way that behaved sanely across DigitalOcean and my local environment. I presume the right Docker image layer caching would address the issue, but I pretty rapidly reached the end of my knowledge and patience.

Dokku's DX does seem great for people doing normal things. :)

I’m in a weird position where I work with both extremely large enterprises and very small startups and not much in between. The enterprises in my area lean towards Azure so it’s still my cloud provider of choice. I haven’t had occasion to look for PaaS providers outside of the major cloud vendors yet. On the startup side, most of these companies want to be cloud first so it’s much easier to start them the right way and avoid the legacy mess that most enterprises deal with today.

Maybe try coolify[0] its provides a better than dokku in my personal experience.

0. https://coolify.io/

I need ARM, is this still the right choice? Nothing is coming up when I hit the site.

Anything of note between the two which you found favorable? I am early-days into designing a Dokku backed system. Dokku has a lot going for it that makes it appealing to me, but happy to learn what I am missing.

The only problem with this is I don’t want something running on my own server that I barely understand. I’d rather use say fly.io that way if there is a problem I can get support from them.

Do you generally move to managed k8s or away from k8s entirely? How is the costing change?

You can still auto-scale. Use dedicated servers for your base load and when sudden demand arises, switch to a cloud provider juggling VMs.

In addition to what has already been said in the other comments, devs have an incentive to push cloud tech against the interest of their employers so they can put the experience in their resume, since cloud tech is seen as more specialized and in higher regard and as such a dev experienced in it commands a higher salary.

Plus who doesn't want to play with the newest, coolest toy on another's dime?

In my experience it's often the other way around. CTO/mgmt/HR would rather swallow the perf/cost hit of Cloud because it makes auditing, compliance, support, dr, security, training and hiring easier. Not to mention it usually provides much more flexibility during the PMF stage of a startup (where product roadmap, team talent/bandwidth and execution speed are often much more important drivers than the raw perf/cost ratio of your infra).

There's definitely an argument to move (some) stuff off cloud later in the journey when flexibility (or dealing with flux/pivoting) becomes less of a primary driver and scale/cost start dominating.

"It's insane not to start your startup on the Cloud and even more so to stay on it" paraphrasing something I think I've read on HN some time ago. But I agree, it's best to treat the Cloud as an intermediate solution and once you stabilize and a clearer picture about requirements is available move at least partially to on-prem (especially the more expensive stuff like compute, storage is I think fine)

I wonder if there is any honest research into the TCO of cloud vs on-prem. In my team we pay ~100k/month for gcp, management doesn't seem to care, since the whole company is on gcp. But they also didn't care when it was on prem and was using hundreds of servers.

RDD or Resume Driven Development

Realistically RDD is a very sane strategy for employees. When I read my company's job ads a few years ago I had serious doubts if I would get hired by them because my resume didn't have the buzzwords they wanted. Since then I always sprinkle in some new tech with each new project, not because it makes sense for the company, but for my resume and future employability.

If you're ever trying to get big enterprise contracts, the kind of scalability, compliance and disaster recovery that providers like AWS/Azure enable are now just table stakes.

Sure, you can get something working on Hetzner but be prepared to answer a lot more questions.

Yes fair point though I doubt some of these super distributed, scalable apps are able to properly execute a disaster recovery... And just because all the serverless components scale "infinitely" doesn't mean the patchwork stitching them together is.

Agreed on your last point of enterprise asking for this which again is just sad that these business "requirements" dictate how to architect and host your software when another way might be the much better one

I run a self-funded SaaS business, so I only deal with real problems, not imaginary ones.

Disaster recovery is a very real problem. Which is why I test it regularly. In a scorched-earth scenario, I can be up and running on a secondary (staging) system or a terraformed cloud system in less than an hour. For my business, that is enough.

The industry lost sight of a lot of things.

We went through a growth boom, and like all of them before, it meant there were lots of inexperienced people being handed lots of money and urgent expectations. It’s a recipe for cargo culting and exploitative marketing.

But growth is slowing and money is getting more expensive, so we’ll slow down and start to re-learn the old lessons with exciting new variations. (Like Here: managing bare metal scaling and with containers and orchestration)

And the whole cycle will repeat in the next boom. That’s our industry for now.

A lot of its the buzz. 'Cloud Hosting' is something everyone thinks they need, despite not grasping why, or that the chances are they arent getting cloud hosting at all, just a VPS.

A solid dedicated server is 99% of the time far more useful than a crippled VPS on shared hardware but it obviously comes at an increased cost if you dont need all the resources they provide.

I think it might be due to the incentives - the costs are borne by the company (thus irrelevant to the employee), however putting new cool technologies on their CV is very relevant to the employee.

> ...company...employee...

Yes - but stupid yearnings to "do what all the cool kids are doing now" are at least as strong in those who would normally be referred to as "managers", vs. "employees".

Good point - and in the case of managers there's also the "nobody got fired for buying IBM" effect:

outcome A) huge microservices/cloud spend goes wrong, well at least you were following best practices, these things happen, what can you do.

outcome B) you went with a Hetzner server and something went wrong, well you are a fool and should have gone with microservices, enjoy looking for a new job.

Thus encouraging managers to choose microservices/cloud. It might not be the right decision for the company, but it's the right decision for the manager, and it's the manager making the decision.

Agreed. I love our physical in house servers, but not knowing AWS in depth really hurt my last job search.

Yeah that's the problem. It's not just a "devs are idiots, they should think of their company and not be obsessed with their CV" - not having worked with the latest trendy tech has a very real cost for developers.

(As does being a consultant wanting an extension and writing software that works, as I found out the hard way.)

> I really don't understand why the industry seems to have lost sight of this.

There's a disconnect between founders and everyone else.

Founders believe they're going to >10x every year.

Reality is that they're 90% likely to fail.

90% of the time - you're fine failing on whatever.

Some % of the 10% of the time you succeed you're still fine without the cloud - at least for several years of success, plenty of time to switch if ever necessary.

Even if you 10x every year, going from year 1 to year 2 you switch from renting two 6-core servers to renting two 64 core servers, in year 3 you rent 20 servers, in year 4 you are renting a small room of racks in a colo space and add a server every other day (or you continue renting servers, if that's your style). Out of all the problems that arise from 10x-ing your organisation every year, this is a really minor one.

The fear.. I'll need 20 servers in 3 years.. the reality: A 4 dollar vps is powerful enough until you hit millions and millions of customers.

would prefer to spend 20-30-50 USD and get a managed deployment infrastructure with rollback, blue/green deployment, github integration and so. this setup can last a long time.

I'd pay good money if you could truly teach me how to do that for recurring costs within a factor of 2 from cheap rental vps/box.

And all that to host what could be served by Hugo static site.

I think one of the reasons is that people confuse physical servers with manual administration. As I said, I do not do manual administration. Nothing ever gets configured on any server by hand. All administration is through ansible.

I only have one ansible setup, and it can work both for virtualized servers and physical ones. No difference. The only difference is that virtualized servers need to be set up with terraform first, and physical ones need to be ordered first and their IPs entered into a configuration file (inventory).

Of course, I am also careful to avoid becoming dependent on many other cloud services. For example, I use VpnCloud (https://github.com/dswd/vpncloud) for communication between the servers. As a side benefit, this also gives me the flexibility to switch to any infrastructure provider at any time.

My main point was that while virtualized offerings do have their uses, there is a (huge) gap between a $10/month hobby VPS and a company with exploding-growth B2C business. Most new businesses actually fall into that gap: you do not expect hockey-stick exponential growth in a profitable B2B SaaS. That's where you should question the usual default choice of "use AWS". I care about my COGS and my margins, so I look at this choice very carefully.

I'll be the counter opinion for this. Let's say you're Sprocket Masters, and you have to run two front ends (a Sprocket Sales site and a Sprocket Repair site) and a backend connecting them. But ultimately your sprockets are made in your factory on-site, and the majority of your staff is related to the manufacture of sprockets.

You're not a software company, fundamentally you make and sell Sprockets.

The opinions here would be hire a big eng/IT staff to "buy and maintain servers" (PaaS is bad) and then likely "write a bunch of code yourself" (SaaS is bad) or whatever is currently popular here (last thread here it was "Pushing PHP over SFTP without Git, and you're silly if you need more" lol)

But I believe businesses should do One Thing Well and avoid trying to compete (doing it manually) for things outside of their core competency. In this case, I would definitely think Sprocket Masters should not attempt to manage their own hardware and should rely on a provider to handle scaling, security, uptime, compliance, and all the little details. I also think their software should be bog-standard with as little in-house as possible. They're not a software shop and should be writing as little code as possible.

Realistically Widget Masters could run these sites with a rather small staff unless they decided to do it all manually, in which case they'd probably need a lot larger staff.

The core business for Sprocket Masters isn't really tech stuff: it's making sprockets. For those kind of businesses it makes a lot of sense; the tech stuff is required, but only in the sane sense that a phone line or electricity is.

However, what I also see – and what I think the previous poster was talking about – are businesses where tech is at the core of the business, and there it often makes less sense, and instead of saving time it seems to cost time. There's a reason there are AWS experts: it's not trivial. "Real" servers also aren't trivial, but also not necessarily harder than cloud services.

"Tech" isn't a thing. If I ran a company that produced CAD software, I would work in "tech", but I would have no reason to believe that my competency in writing CAD software should translate to competency in maintaining web servers or network infrastructure.

You still have to do all that with cloud though - instead of firewalls and IP tables it’s VPCs, IAM, security groups, auto scaling groups, etc etc. The reason I think it’s still probably better for Sprocket Masters to go cloud is purely because those skills are easier to hire for.

Sounds like Sprocket Masters should really outsource their two sites to a an agency that specialises in maintaining such solutions for SMEs, then they wouldn't need to concern themselves with the nitty gritty of hosting/backups/security/updates etc…

Sounds like Sprocket Masters should really outsource their two sites to a an agency that specialises in maintaining such solutions for SMEs...

But those agencies don't want to have to staff for maintaining physical hardware either...

> . In this case, I would definitely think Sprocket Masters should not attempt to manage their own hardware and should rely on a provider to handle scaling, security, uptime, compliance, and all the little details.

But Sprocket Masters still has to have an expensive Cloud Consultant on retainer simply to respond to emergencies.

If you're going to have someone on staff to deal with the cloud issues, you may as well rent a server instead.

When the web was starting, that was the hardest sell to companies. They don't want to worry about those technical staff they just want a website/ecommerce working and not putting too much resource on it. Then social media happens and those sites just became status symbols or just to retain the email address.

it's not all or nothing. There's plenty of cases where using AWS or similar makes sense, but using the cloud for absolutely everything has a bunch of costs, both visible and invisible. Nuance and taste are the hardest things to teach, and these are in short supply in many engineering orgs.

Because cloud is the new "No one gets fired for using IBM". If you are using dedicated servers somewhere, you cannot have the same reputation of say Using AWS for cloud. Everyone knows the AWSs, Google Cloud etc and no one gets fired for selecting/using them.

I personally have run Dedicated Servers for our business in the earlier days but as we expanded and scaled, it became a lot easier to go with the Cloud providers to provision various services quickly even though the costs went up. Not to mention it is a lot easier to tell a prospective customer that you use "cloud like AWS" than "Oh we rent these machines in a data center run by some company" (which can actually be better but customers mostly wont get that). Audit, Compliance and others.

> trivially solved with one or two dedicated servers...

For many even lot less than that. I run a small side project[1] that went viral a few times (30K views in 24 hours or so) and it is running on a single core CPU web server and a managed Postgres likewise on a single CPU core. It hasn't even been close to full utilization.

1: https://aihelperbot.com/

I feel like many people use the cloud "wrong". If you're just standing up VMs then yeah, it's going to be slower and more expensive than renting a physical server. If you're just going to setup VMs and pretend you're still on-prem you're better off not switching. But I've got projects setup on PaaS services that literally cost a few bucks a month because it's only charging me for actual use and not idle time. I've got some small customers (tens of thousands of users) who I've helped move into the cloud who see the same thing. They have SPA websites backed by lambda functions and on-demand databases. If their website isn't being hit, it's not costing them anything at all. The only lock-in is the boiler plate code around the lambda activation which could be switched fairly easily to target any Functions as a Service platform.

I'm curious to know what on-demand databases do you use with AWS Lambdas? Two options that I've evaluated are: Aurora Serverless v2, which isn't quite serverless in practice and DynamoDB. And the mental model for DynamoDB very different from that of traditional SQL databases.

I’m actually primarily building on Azure with Cosmos DB. The mental model is very different, but I think important if you want to build cheap cloud services. I think that’s the larger point. You have to think about doing things differently in the cloud otherwise you’re just deploying and managing VMs as if you were on-prem albeit with more scaling control. It feels like everyone who is complaining about how expensive cloud hosting is go into it thinking it’s just someone else hosting the VMs where as a VM is the option of last resort for me.

I am responsible for about 10-12 t3.medium-class servers (number includes redundant servers) and the services running on them. All of them are fairly overprovisioned for what they are, excepting that the monitoring system we use on them for some reason likes to consume about half-a-gig of RAM so I can't run on even smaller servers. I write my services in Go, so the CPU isn't sitting there spending 98% of its time chasing pointers and can use multiple CPUs as needed for its heavy-hitting tasks. It's a rough day for those servers when they pass 5% average CPU for the day.

Could I switch some of them to lambda functions? Or switch to ECS? Or switch to some other cloud service du jour? Maybe. But the amount of time I spent writing this comment is already about six month's worth of savings for such a switch. If it's any more difficult than "push button, receive 100% reliably-translated service", it's hard to justify.

Some of this is also because the cloud does provide some other services now that enable this sort of thing. I don't need to run a Kafka cluster for basic messaging, they all come with a message bus. I use that. I use the hosted DB options. I use S3 or equivalent, etc. I find what's left over is almost hardly worth the hassle of trying to jam myself into some other paradigm when I'm paying single-digit dollars a month to just run on an EC2 instance.

It is absolutely the case that not everyone or every project can do this. I'm not stuck to this as an end-goal. When it doesn't work I immediately take appropriate scaling action. I'm not suggesting that you go rearchitect anything based on this. I'm just saying, it's not an option to be despised. It has a lot of flexibility in it, and the billing is quite consistent (or, to put it another way, the fact that if I suddenly have 50x the traffic, my system starts choking and sputtering noticeably rather than simply charging me hundreds of dollars more is a feature to me rather than a bug), and you are generally not stretching yourself to contort into some paradigm that is convenient for some cloud service but may not be convenient for you.

> Even the mere suggestion of renting a dedicated server provokes scorn from devops teams.

Have you ever had to manage one of those environments?

The thing is, if you want to get some basic more-than-one-person scalability and proper devops then you have to overprovision by a significant factor (possibly voiding your savings).

You're inevitably going to end up with a bespoke solution, which means new joiners will have a harder time getting mastery of the system and significant people leaving the company will bring their intimate knowledge of your infrastructure with them. You're back to pets instead of cattles. Some servers are special, after a while automation means "a lot of glue shell scripts here and there" and an OS upgrade means either half infra is KO for a while or you don't do OS upgrades at all.

And in the fortunate case you need to scale up... You might find unpleasant surprises.

And don't ever get me started on the networking side. Unless you're renting your whole rack and placing your own networking hardware, you get what you get. Which could be very poor in either functionalities or performances... Assuming you're not doing anything fancy.

> The thing is, if you want to get some basic more-than-one-person scalability and proper devops then you have to overprovision by a significant factor

If you want 100.0000% uptime, sure. But you don't usually. The companies that want that kind of uptime normally has teams dedicated to it anyway.

And scaling works well on bare-metal too if you scale vertically - have you any idea the amount of power and throughput you can get from a single server?

It's concerning to keep hearing about "scaling" when the speaker means "horizontal scaling".

If you requirements are "scaling", then vertical scaling will take you far.

If your requirements are "horizontal scaling on demand", then, sure, cloud providers will help there. But, TBH, few places need that sort of scaling.

Just getting a highly available storage with any kind of performance requires you to overprovision by something like 12x (3 boxes with raid 1+0) which is frankly insane. And then you have to manage it and fix it when it inevitably breaks. The team who’s supposed to do that will be swamped by trying to keep the other 15 infrastructure services running and upgraded to a version which is only barely out of vendor support.

> Just getting a highly available storage with any kind of performance requires you to overprovision by something like 12x (3 boxes with raid 1+0) which is frankly insane.

I'm not saying 100% uptime on bare metal is cheap, I'm saying 100% uptime is frequently not needed.

If only the config above gave 100% uptime... far from it IME

>I really don't understand why the industry seems to have lost sight of this.

Because the industry is full of people who are chasing trends and keywords and to which the most important thing is to add those keywords to the CVs.

Its big companies - their priorities are different. Saving a couple million dollars isn't worth the tradeoff of not being industry standrd (AWS, Google, MS). They hire tech consulting companies who sell their product, large deployments.

"No one ever got fired for buying AWS"

You mean Oracle. That's another full circle right here...

> but highly scalable

IME aiming for scalability is exceedingly wrong for most services/applications/whatever. And usually you pay so much overhead for the "scalable" solutions, that you then need to scale to make up for it.

devops people want things to be expensive and complicated to justify their salaries

Having talked to AWS evangelists, it seems that they truly believe it cuts costs. In their reality this is true, because an alternative with the same guarantees and risk costs more. The implicit disagreement is that you often don’t need all those guarantees.

> because an alternative with the same guarantees and risk costs more

I really doubt this. The one theme you see on almost every AWS proponent is some high amount of delusion about what guarantees AWS actually provides you.

Hmm. I guess there's guarantees that it won't happen, and guarantees that if it happens then it wasn't supposed to and can't be your fault, and the latter may still be pretty attractive to some decision-makers.

> if it happens then it wasn't supposed to and can't be your fault

Yeah, sure. Nobody gets fired for buying from IBM.

Well, if large companies had any competence in decision-making, they would be unbeatable and it would be hopeless to work on anything else at all. So, yeah, that's a public good.

Anybody with experience on the ops side of that is absolutely in favor of simple and easy to understand -- and that often gets you cheap as a side effect.

Source: almost thirty years of ops.

In my experience those devops people where hired and empowered by senior managers to make things expensive and complicate because the company is 'enterprise' and has to follow "industry best practices"

Remember how Twitter lost 80% of its staff and still kept running?

And Twitter ceo turned his account private because they can’t figure out what the algorithm does

For now... There's a lot of inertia obviously.

Nice generalization.

So far I haven’t noticed that if I spend more for the company that I also get paid more.

if you need durability in the cloud you just pick any object storage service from the 3 bigs, and a db with multiple deployment with automatic replication and managed failover.

getting the equivalent reliability with irons is a lot more expensive than renting "two dedicated servers" - now you might be fine with one server and a backup solution, and that's fair. but a sysad to create all that, even on a short contract for the initial setup and no maintenance, is going to go well beyond the cloud price difference, especially if there's a database in the mix and you care about that data.

I'm not saying there's no place for cloud, but that many workloads benefit from using dedicated hardware. It is often much simpler to deploy to a dedicated server rather than spending months rearchitecting your system (or even worse, cutting customer facing features!) to efficiently make use of many small cloud instances.

There are several reasons for that: - lack of knowledge how to maintain own servers on premises - MASSIVE PR by a handful of top tier cloud providers to instill the notion that cloud is the only solution, the best one and most advanced technologically and attractive silk set wise - hockey stick growth expectations of (past days) startups from investors, forming the mindset that it must be met from day one in deployment - “no one got fired for buying IBM” managers/tech leaders mentality, where IBM is replaced with “cloud” - difficulty obtaining true costs information of cloud vs on premises deployment. For example - a team of ~16 people already pays off the costs of dedicated sysadmin to maintain on premises servers - etc.

As a developer I appreciate the compartmentalization, being able to make truly isolated changes. This is something that's not trivial for lots of folks, me included. I guess if one has the expertise in house to manage a K8s cluster on a raw machine, that'd be an option but I, personally, am not gonna touch that with a ten foot pole. So yeah with a big and capable enough infrastructure team that complexity could be abstracted away but in reality I (where I've worked) haven't seen that yet.

It feels like artificial scarcity similar to what you see in high fixed cost monopolies. And that is exactly what most corporate IT departments are.

Growth hides sins. I ran a large self-funded enterprise service at a loss for 4-5 years, which is possible if you are growing 30%.

Today, cloud is similar - the time to market is quicker as there are less moving parts. When the economy tanks and growth slows, the beancounters come in and do their thing.

It happens every time.

the most funny thing is that cloud engineering teams have no problem of using EC2 hosts, but somehow don't want to rent dedicated servers/VPC from regular datacenter companies.

This only makes AWS richer at the expense of companies and cloud teams

> the most funny thing is that cloud engineering teams have no problem of using EC2 hosts, but somehow don't want to rent dedicated servers/VPC from regular datacenter companies.

It is trivial to provision and unprovision an EC2 instance automatically, within seconds if your deployment needs to scale up or scale down. That's what makes it fundamentally different from a bare metal server.

Now, I'm not denying that it might still be more cost effective when compared to AWS to provision a few more dedicated servers than you'll need, but when you have really unpredictable workloads it's not easy to keep up.

the whole concept of a need to spin up more VM instances to meet increased load is flawed and is driven by cloud vendors to make money.

if you are spinning up and shutting down VMs to meet demand curve - something is seriously wrong with your architecture.

Did it ever occur to you, how come stackoverflow uses ~8 dedicated servers to serve entire world and doesn't need to spin up and shutdown VMs to meet global customer demand?

When planning compute infrastructure, it is important to go back to basics and not fall for cloud vendor's propaganda

> serve entire world and doesn't need to spin up and shutdown VMs to meet global customer demand?

You said it yourself. Not all applications need to serve entire world, therefore demand will be lower when people go to sleep.

Even with global applications there are regulations that require you to host applications and data in specific regions. Imagine an application that is used by customers in Europe, Australia, and the US. They need to be served from regional data centres, and one cluster will be mostly sleeping when the other is running (because of timezones). With dedicated servers you would waste 60-70% of your resources, while using something like EC2/Fargate you can scale down to almost 0 when your region is sleeping.

Anybody tried mixing cloud + self hosted k8s on hetzner or is it too complicated to do?

You'd probably have some latency issues, plus waste a ton of $$$ on the bandwidth egress fees.

You've already lost if you install k8s.

> Even the mere suggestion of renting a dedicated server provokes scorn from devops teams.

There is a method to the madness, here it is called "job-security-driven development".

> Even the mere suggestion of renting a dedicated server provokes scorn from devops teams.

Because it's a threat to their jobs.

I just want to go back to leasing an always-available Opteron with optional remote hands.

Reply on news.ycombinator.com