Hacker News Reader

I worked at several startups before working at FAANG - none of them operated at any scale that they needed to be on AWS. None of them ever grew enough to need to be on AWS either.

One of them did a bunch of stuff that made use of AWS - but it didn't deliver any value to our customers, and probably made our product substantially worse. It cost $30k a day, and unquestionably was not worth the cost, and everyone on the team was fired after a few months.

I worked at a startup that didn't use the cloud and was hemorrhaging money by only using a handful of compute in a DC despite renting a whole bunch of space/power. They could have moved to AWS and saved a ton on cost, but their CTO was comfortable with renting colo space and knew how to run a colo-based tech company. There's not much you can do when people make bad decisions. At most you can understand the tradeoffs of the decision space and do your best. There's rarely a silver bullet.

There are options beside AWS and colo.

A lot of companies are fine just using a handful of VPSes.

I think the biggest problem with starting on AWS is engineers are extremely susceptible to playing with new toys - for better or worse.

Engineers are going to try all AWSes new stuff, which is fine. The problem stems from engineers wanting to do stuff on the newest shiniest AWS toy whether that actually makes sense financially or much more importantly in terms of the product roadmap.

> whether that actually makes sense financially or much more importantly in terms of the product roadmap.

If design docs don't at least mention the cost - and your business cares about cost - then you should probably have a chat with the engineering dept.

At a prior FAANG job we included basic cost analysis in our decisions. You don't always need the cheaper option if it doesn't make sense, but you should probably know the traffic that shiny new toy will receive and the cost associated with it. (eg. soln A needs 25% more compute, and the internal price is X vs soln B needs 10% more compute, but will increase storage costs for caching at X per gb for ~Y gb).

I recall a case where a coworker (Senior SDE in Silicon Valley, so easily 25k a month salary) spent a month reducing our $500 SQS bill to $350 because our director wanted to see teams making smart financial optimizations. Meanwhile, our EC2 bill was $50k a month. Not everything is smart financially to focus on fixing. What is the compSci quote on premature optimization?

It's "premature ejaculation is the root of all evil"

Yeah there's tons of options. There's the big clouds/cloud-native stuff. There's the cloud-VMs/VPSes. There's colos. There's making a DC at an office through DIA. Understanding these options is key to making a good, informed decision IMO.

The advantage of AWS for a very small organization is flexibility and convenience, not scale.

Of course, if you can't trust your developers to spend money wisely, you need guardrails on cost, no matter where you're spending it.

Flexibility and convenience are benefits of the cloud in general, not exclusive to a single PaaS like AWS.

FAANG doesn't need to use cloud either. Notably, Facebook doesn't (and, a bunch of stuff at Amazon was still not on cloud until _long_ after AWS was invented).

Facebook doesn't use the cloud? Are you crazy? They built their own cloud. They just don't sell it as a service.

Sure - not everything at these companies needs to be on the cloud. But when you're dealing with sensitive information, you don't want people rolling their own infrastructure.

Is it really cloud if you built it yourself and run it on your own hardware in your own data center?

Certainly! I joke that I have my own private cloud: I have a k8s cluster running on 6 machines in my closet datacenter.

When you're one of the largest data center owners in the world, its pretty close? Its not like they're colo'ing? How do you define cloud?

Yes! the fact you are asking this belies your understanding of the cloud.

Can you elaborate on what your understanding of the term is?

Well then I guess everyone is actually already in the cloud. They just have varying levels of UI/UX.

Yes. A private cloud due to the tech stack at work.

At the end of 2021 Facebook announced they would begin to use more AWS services [0]. Now, I wouldn’t call it Facebook going full cloud and I definitely expect FB to continue to use on-prem for the majority of their compute/storage; but I think it’s worthwhile to note that even FB (which may be one of the largest private clouds) sees value in using cloud providers to supplement their on-prem solutions.

[0] https://press.aboutamazon.com/news-releases/news-release-det...

This is some peripheral or business continuity-type stuff. FB will obviously not move any of their critical stuff there.

Isn't Netflix 100% on AWS?

IIRC Netflix run their own boxes for content delivery, which makes sense because of the exorbitant cost of bandwidth on AWS.

Reminds me of these YouTube videos filled with bad advice for newbies. For example, "Why I never code with ELSE statements, and YOU shouldn't either". It sounds profound to a newbie, but it's all just disingenuous dogmatic click bait.

I know you're kidding, but there are legit videos saying you should never use If statements

I wish I was kidding, but that sounds even worse!

- https://www.youtube.com/watch?v=kzuCnivRoEw

- https://www.youtube.com/watch?v=_ougvb8mT7k

- https://www.youtube.com/watch?v=EumXak7TyQ0

Except...these are right. Minimizing nesting, and setting up validation failures as exit-early conditions from functions to have the main function body focused on the happy path are absolutely valid principles of clean code.

Obviously there are always exceptions, but as a principle you could do a lot worse.

FWIW I rarely use else statements and prefer early returns. Might be trauma from dealing with legacy code where you'd need an ultra wide monitor to get the end of the indentation.

Just to add here, even aside from these trade-offs...

The main alternative is: host the hardware yourself, as it might be enough.

Cool. How many people here are also great sysadmins? Probably a very small number. That's not really an alternative. And furthermore, most of the counter point doesn't really exist.

They just mainly pointed out that "yeah, devops is hard, and lots of things are done by devops that seem simple but arent" but they didn't make an argument that aws is somehow an overnengineering thing that you can avoid if you would only do XYZ and remember that you probably have small scale.

> How many people here are also great sysadmins? Probably a very small number.

I am. And there probably aren’t that many around here due to the extreme prejudice and derision Developers direct towards Sysadmins. We’re apparently a bunch of low-skill knuckle-dragging hardware monkeys, while at the same time able to do things so difficult that no developer can figure it out so they just go to the cloud instead “to avoid learning all that stuff”.

It will take much less time to become a decent sysadmin of your startup server than to become a decent AWS DevOps person

No, hosting the hardware yourself is not the main alternative.

Very few non-cloud users actually host the hardware themselves. You can rent dedicated servers or even VMs just about anywhere. The hosting company manages and maintains the hardware as part of the monthly price.

A lot of small sites can probably do great without a sysadmin, nor AWS. If you think about it AWS can be a pre-mature optimization when just running your site off a Rapsberry Pi in your basement will do.

How is a self-managed raspberry pi in your basement in any way better than using a t2.nano on the free tier?

Heh, technically t2 instances are burstable CPU so you can't control when you'll be pushed off CPU time. With a RasPi you have full control of the compute. Of course, the t2 has faster transit to pretty much any IX, is in a DC with a UPS, has an SLA for staying up, etc. so from a networking perspective it's leaps and bounds better. Depends on if you need the compute or not.

Right, so if and when you determine you actually need more compute, it's trivial to bump the instance type. Not so much in the basement, seems pretty pointless.

If you mean re-attach your EBS storage to a bigger instance type, you can just remove the SD card and insert into a bigger computer (or clone it to a hard drive).

You don't need to manually manage EBS. Shutdown instance, change instance type, boot.

I guess you could buy raspberry pi with more memory (but not more cores) and swap the sd card, but beyond that you are stuck. aarch64 rootfs will not boot on commodity x86 hardware. sd card performance and reliability is many times worse than EBS anyway.

Put nginx in front of a bunch of Pis and load balance them! Just clone the SD card and distribute to a bunch of Pis. I mean, you'll have to scale horizontally at some point even on EC2 and it's roughly the same complexity. If the value prop of EC2 is hardware abstraction, I don't think it offers that much over Raspberry Pis, unless you are going for higher performing hardware.

This feels disingenuous, like pushing a DIY narrative, similar to what the OP is doing. Whether you think it offers value or not, a Rasberry Pi at home is _far_ from a cloud VPS. A cloud VPS is behind a net connection with an uptime SLA, has redundant power, can have an SLA of its own, is not NATed, has a fast NIC, and usually is low latency to transit to because of presence in an IX. If you're equating the two, you don't understand why datacenters exist. You can _reject_ this value, but you'd need to be clear about what your expectations are out of then IMO. It's fair to make a comparison between a coloed host or a VPS in another cloud service (like DO or Hetzner) with AWS because it comes with most of the above things (just usually worse transit), but to deny that there's value in a datacenter feels like willful ignorance.

Just FYI residential ISPs are absolutely terrible. Even my ISP, a FTTH one, has uptime lower than 99%. This is aside from layers of CGNAT and your own home equipment's SLA. If you're willing to run a product on this kind of RasPi infrastructure feel free, but don't claim there isn't any value in a datacenter.

My point is that a lot of people don't need all those things. As the article points out, a managed dedicated server offers most of those benefits and the redundancy plus "easy scaling" of AWS is not even necessary if your goal is just to avoid debugging hardware issues.

It's easy to setup a home raspberry pi. It's easy to go one step up and setup a colocated raspberry pi. It's easy to setup a managed dedicated server. Auto scaling virtually provisioned hardware is great for a big company like Netflix with actual daily fluctuating demand. Most people don't need such advanced scaling features.

I previously had Sonic.net, which was one of the best fiber offerings for consumers available. 1 gbps up and down, unmetered, and was pretty much never down during the period I had it.

Maybe I don't want to learn Raspberry Pi to start my little SaaS idea. To search and click in the AWS console to get some MVP up and running cannot be beaten that easily. Can I get a Raspberry Pi site with a few 100$/month, my efforts included?

I'd argue, for an average software engineer, setting up a Raspberry pi is going to be easier than setting up an EC2 instance. It almost sounds like you've never actually done the "click in the AWS console to get some MVP" running before.

Sorry if this is too harsh, but if you can't figure out how to run an app on a Raspberry Pi then I definitely would't trust you to click around the AWS console.

If I use the Raspberry Pi browser to deploy via the AWS console does that count?

Tradeoffs. If you can run your service on a residential internet SLA (or lack thereof) with residential CGNAT, and you have the ops chops to maintain Linux installs yourself, then do it. If you can run your business with a business-consumer SLA do it. AWS is much more reliable than that.

Realistic AWS non-cloud alternatives are either colo-ing in a DC, using another semi-cloud provider (OVH, Linode, Hetzner, et al.), or buying a DIA circuit for your office and running your own servers from there.

Well, you still have to patch and secure your OS on EC2. As far as EC2, AWS is really just taking over the hardware portion and making it somewhat easier to scale. But rapidly scaling is not something most apps do, even big ones.

If we're talking about serverless then I think that containerization (either running containers on bare metal, or on kubernetes) changes the value prop of serverless a lot, because if you've got a container environment you can easily just clone an off-the-shelf production-ready container to deploy your app.

As far as your host OS goes get a production ready image to run your production ready container images. It's a one time thing. Keeping the OS updated? This is not brain surgery every time there is an update. Plus you can configure it to automatically install security updates.

> Well, you still have to patch and secure your OS on EC2. As far as EC2, AWS is really just taking over the hardware portion and making it somewhat easier to scale. But rapidly scaling is not something most apps do, even big ones.

You're stuck on this theme that AWS is somehow akin to scaling. That's one benefit of AWS but it's not the only one. At small scale the margins on AWS are peanuts. A t4g micro is $6.15 / mo. The equivalent on Digital Ocean is $5 / mo. Buying your own Raspberry Pi 4 would be ~ $70 with an enclosure/peripherals, so you'd break even at ... 14 months of running your t4g. This isn't counting the power used (which would probably be minimal on a Raspberry Pi.) That overhead is nothing.

> As far as your host OS goes get a production ready image to run your production ready container images. It's a one time thing. Keeping the OS updated? This is not brain surgery every time there is an update. Plus you can configure it to automatically install security updates.

There's more to it than that. You're just thinking about running software not how packets get from a user's machine to your running software. Most residential connections don't come with a stable/static IPv4. You can update a DNS entry with your changing IP, but then you're down for however long it takes you to change your A record and however long your domain's TTLs to expire. If you pay for a static IPv4 then you've already paid for more than what you're getting from a cloud VPS. Then there's the fact that residential ISPs block tons of ports, have no SLAs on uptime, can drop your traffic without warning or recourse, etc etc.

If you're running a tiny, mostly-static site with minimal uptime requirements then you'll pay less and spend much less effort using a shared webhosting platform. They'll do all the ops for you and you get charged peanuts since these providers usually colo their own machines and run hundreds of sites on them. Dreamhost can serve a Wordpress site for $1.99 / mo with no ops work required. That pays for 35 months of running a Raspberry Pi.

A Raspberry pi is $35, for the latest model, and you definitely do not need an enclosure. You do need a power adapter, which may be around $10. But also, an older Raspberry pi can be even cheaper!

I'm not advocating for people hosting sites on a Raspberry pi in their homes, but it's certainly easy to do and if your operation is small enough and you have an extra computer lying around the cost is actually near $0.

Pretty much every ISP I've used in the US has had mostly static IPs. They usually didn't change unless the modem got rebooted. This is good enough for your minecraft server or unimportant personal website.

But of course if your residential internet is no longer serving your hosting needs, you can go a step up and get a virtual host or colocate your old computer! Yes colocation will cost a lot more than a virtual host on a PHP shared host, but you also get more computing bang for your buck.

Sysadmin-ing is not the boogeyman it once was!

So don’t use EC2 either. Or anything that should be managed by a sysop

Complex uncapped pricing is not a theoretical problem. Cloud lock-in is not a theoretical problem. Loss of control in case of downtime or a "ban" is not a theoretical problem. Cloud reducing complexity is questionable and may swing the other way.

They are valid concerns or at least warnings. You're obviously right that there's tradeoffs but the thing I'm sensing in our industry is that it's increasingly seen as just the default "thoughtless" infra model.

It's simply a predictable swing of the pendulum. 5-10 years ago the same formula would play out except then, it was AWS solving all your problems, and how hardware is so hard because apparently you need to hire 3 full time employees because machines run on sysadmin tears rather than electricity so of course you need someone constantly lording over your 10 servers.

>with some hand-wringing about complexity (which is relative to the person

Are you sure it isn't relative to other solutions? I mean sure, you can learn it well enough that it's not complex for you anymore but there is still some objective measure of its complexity that relates to how difficult it was to learn.

IMO a good example here is ops. I grew up in the era where net businesses started by renting colo space or getting a T1 line. When I learned about starting net projects, I learned how to set up all the things needed to get a project going in one of these settings. To me, running ops on a Linux or BSD box is simple; I've literally been doing it since I was a kid. These days it's a lot harder to find people with this expertise. To _me_, buying a DIA circuit and setting up a mini-DC for a business is simple. To most others who experienced Linux through Ubuntu or mostly grew up using MacOS, using cloud-VMs is probably a lot simpler.

To come from the other direction, not every app is or will ever need to be running at Google scale. Most of what AWS offers just triggers large enterprise warm fuzzy feelings. Even when they're totally not needed.

Not to mention the lock in...