Hacker News Reader

Comment by southerntofu | Hacker News Reader

Just did the calc and 600GB/day is about 55Mbit/s. That's really not a lot and if there's not too much computation server-side you could serve this from a raspberry pi at home (provided you have good uplink). But that's assuming you keep the CloudFlare cache of course, or as author mentioned himself, advertising only torrents for the multi-gig files.

I really don't understand the cloud craze. Everything is more complex to debug, more expensive, and more shitty in all the possible ways you can imagine. I mean i was not exactly a fan of the VPS craze 10-15 years ago, but at least it wouldn't automatically ruin your bank account whenever you got a little traffic.

Kudos to the author for having so much money (thousands in one month?!) to waste. I wish i did too :)

View on HN

> Everything is more complex to debug, more expensive, and more shitty in all the possible ways you can imagine.

Coming from traditional infrastructure and development methods, you're mostly right. Part of the expectation of the cloud is that you do things _their way_. And even then each cloud provider does things a little differently. However, if you're willing to subscribe to the <insert provider> way of doing things it (and you'll have to trust me here) makes many things easier. Here's a short list:

* networking setup is free/cheap/doesn't require a Cisco cert. you can trust a developer to set things up.

* object storage is so much easier than any file hosting scheme you can come up with

* the path from container-on-a-host to container-in-a-cluster to container-in-{serverless,k8s} is extremely straightforward

* I turn all my dev/test servers off at night and they don't cost me a thing

* consumption based compute will result in a much cheaper solution than a VPS or colo (admittedly there are many assumptions baked into this)

* some core services (like sqs, sns on Amazon) are extremely cheap and have provably reduced development time because you're not having to build these abstractions yourself.

This all being said I'm not advocating an all-in approach without thinking it through, but to do so where it's easy and makes sense.

EDIT: clarity

> networking setup is free/cheap/doesn't require a Cisco cert. you can trust a developer to set things up.

Bare metal hosts set up the network for you. You may need to know how to configure a local network interface. Even if you actually rack and stack many colos will give you a drop with network set up. You don't need to do what you describe unless you are building your own DC.

> object storage is so much easier than any file hosting scheme you can come up with

That matters if your data volume is truly massive. Only a small percentage have this problem. Also AWS inbound is free so you could upload big data to AWS and warehouse it there if you wanted. Not using big cloud for everything doesn't mean you can't use it for anything.

> the path from container-on-a-host to container-in-a-cluster to container-in-{serverless,k8s} is extremely straightforward

This is the one spot where admittedly you will have to spend more in administration. You'll need to either run your own k8s or Nomad or adopt a different configuration, and you may have to think about it a bit more.

> I turn all my dev/test servers off at night and they don't cost me a thing

You could still do this. Just host live somewhere else. You could also test on a local VM, which is what we do. Obviously that depends on how big your app is.

> consumption based compute will result in a much cheaper solution than a VPS or colo (admittedly there are many assumptions baked into this)

You only see the savings if they are passed onto you. What we've seen is that Moore's Law savings have not been passed on by cloud providers. Look at what you can get at a bare metal host compared to how much the same compute costs in cloud. Years ago the difference would not have been so large.

Bandwidth costs in cloud are insane, and most use asymmetric pricing where inbound bandwidth is free. This is known as "roach motel pricing" for a reason. Data goes in, but it doesn't come out.

> some core services (like sqs, sns on Amazon) are extremely cheap and have provably reduced development time because you're not having to build these abstractions yourself.

Fair, but they make their money back elsewhere. Those are lures to get you locked in so you now have to pay their crazy compute and bandwidth egress charges.

Here's an example. There are more.

https://www.datapacket.com

Haven't even heard of DataPacket, thanks for the link!

And yeah I agree about the "some services are super low cost so you get hooked" thing. Always been my impression of Amazon: they look for what they can apply scale savings on (usually object storage, it seems) and make it cheap and then over-charge for almost everything else.

AWS is the new Oracle.

That's not really it.

The funny business in Amazon's pricing is their Egress Bandwidth, everything is rational.

You're looking at the pricing from a 'cost plus' perspective which is not generally how things are priced.

AWS core use case is IT departments being able to offload all of their infra.

It's a massive, massive advantage. It's so, so much easier and more flexible to use AWS that there is no comparison. It's a 'no brainer' from a cost perspective, which is why, cost usually isn't a barrier with AWS.

Cost only becomes a primary issue when the margin of AWS services is reflected in the cost of the product itself, i.e. when you are hosting a lot of content.

So if you are Phizer, and your IT department uses AWS, the cost is irrelevant.

If you are Dropbox, selling storage for $X/Gigabyte, and your competitors are reducing their prices and you're giving all of your margin to AWS, then you have to do something, i.e. 'make your own infra'.

I mean OK but I've been in big corps and they end up hiring a ton of DevOps that basically specialize in AWS.

Is that still cheaper? When you have 30+ very well-paid dedicated DevOps specialists? Maybe it is, I am just skeptical while looking at it as an outsider and without solid data.

Datapacket shows Discord on their customer list, I didn't know discord used VPS / Bare Metal or is it like they just tried it once and Datapacket struck their name to their landing page ?

Have you used DataPacket? If so, how's their uptime? Do they have any sort of automated failover so your service doesn't go down if something happens to a single box or rack?

ZeroTier has stuff at DataPacket. It never goes down. We've had 1 year plus uptimes and the network is rock solid.

> I turn all my dev/test servers off at night and they don't cost me a thing

I bet it still costs you more than my Hetzner one despite me not having to care about turning it on or off. I mean it's great that the cloud gives you this flexibility but you wouldn't need it to begin with if it wasn't so expensive.

When you are growing, it’s a no brainer. When you are at steady state it depends.

As a case in point, I worked in standing up a critical system in a large enterprise a few years ago. We spent about $12M on compute, storage, networking, etc. At operational state, it was about 40% cheaper than AWS. The problem is, it all sat there for 6-18 months filling up before we fully hit that state.

With a cloud provider, you pay a high unit cost but if you engineer intelligently your costs should move with utilization. Except for government, most entities generally want to see opex move with revenue and prefer to minimize capex where possible.

You're an order of magnitude larger than what I work on, but on our last big project we purchased and installed half in the first year, then the remaining half 18 months later.

Keep in mind that size tends to lower intelligence! ;)

The cloud is great for scaling. The lead time for new servers deployed in a data center is weeks compared to seconds in the cloud. Plus there's no sunk cost in the cloud - you can turn it off when done and it evaporates.

Also, the cloud offers managed software as a service. You don't have to manage your own HA DB cluster or PubSub. It's all just there and it works. That can save you a lot on technical labor costs.

But yes, I do agree with your point. If you don't know what you're doing, you can nuke your budget super quick.

The technical ability to scale is a bit meaningless if you can't afford it.

Then that becomes part of your business + technology planning conversations:

"This is the cost of scaling, this is the cost of owning our own infra, how does that fit into our budgeting and requirements?"

"If you can't afford it" is doing a lot of assuming and a lot of heavy lifting in that statement. Whether or not you can afford it depends strongly on your scaling bounds (how much you need to scale) and how you've chosen to implement it.

There are plenty of tools and systems that can present a sufficiently linear cost relationship to load and usage that, should your COGS versus revenue make sense, the marginal cost of increased cloud resources a no-brainer--especially versus always-paid-for hardware. If you don't have such a linear relationship you're as much in the position of deciding whether the project is viable as you are anything else.

> If you don't know what you're doing, you can nuke your budget super quick.

And even if you do, which I think you’ll agree Troy Hunt does.

The cloud is great for scaling indeed, but a cheap Intel V4 server with 44 cores from Ebay for $2000 can handle a shit ton of traffic too.

If I were building a new business, I would use both cloud and colo. But I do understand that everyone don't have that luxury.

If you have a large environment to build in a certain region, the cloud lead time is months also. We have to give our cloud provider months notice before building in a region. But we have a pretty serious and profitable workload. Your statement is correct for the 90% of companies with relatively small infrastructure needs.

> The lead time for new servers deployed in a data center is weeks compared to seconds in the cloud.

The lead time for Hetzner or OVH is measured in minutes and is appropriate for the majority of use cases. Old-school providers like these used to run the entire internet before the cloud craze started.

Colocation & own datacenter is not a good choice for most startups. Seems like a lot of people here miss a step and only go from one extreme to the other. There's a middle-ground of renting bare-metal that gives you amazing price/performance with none of the drawbacks of colocation or running your own DC.

"I really don't understand the cloud craze"

The opposite, I don't understand why anyone would ever put up a server if they didn't have to.

It's not 'processing power' that's going to be the 'big cost' for most projects.

It's headcount and salary.

If you can materially improve the operating ability of your company, then a few $K in cloud fees is dirt cheap.

I used to work at a 'tech company' that made a physical product and our IT was abysmal. We had to wait weeks for our sysadmins to order blades, get things set up, there were outages etc..

If a project is definitely going to be 'a few linux servers and never more' - even then it would be cheaper and more reasonable to use virtual instances.

The time to 'roll your own' is when the infra. operating costs are a material part of your business.

For example, 'Dropbox' invariably had to roll their own infra, that was inevitable.

Similarly others.

That said - as this article indicates, it's easy to 'over do it' and end up in ridiculous amounts of complexity.

The Amazon IAM security model has always been bizarre and confusing, and the number of AWS services is mind-boggling.

But the core case of EC2+S3 +Networking, and then maybe a couple of other enhanced services for special case works fine.

I also object to what I think is a vast overuse of Cloudflare, I just don't believe that in most scenarios needing to have content at the edge really changes the experience that much.

> It's headcount and salary.

This only really applies to fully-managed services such as Heroku. Every other cloud still needs a DevOps person according to my experience in many companies.

Yes, but one cloud devops can do what 10x what sysadmins, hardware and network engineers can do.

Just security alone, in terms of managing access to all of those resources, various forms of backup, across regions ... it's just out of reach for most organizations.

> 600GB/day is about 55Mbit/s

In what universe? This frictionless perfect vacuum where traffic comes in a wholly predictable consistent continuum?

Good point, it's just an average. And to be fair i checked the numbers in the article it seemed closer to 3.2TB/day which is closer to 300Mbit/s. But what i meant is a home fiber connection can deal with that. Although consumer ISPs don't have good bandwidth over all routes (it's good to Youtube/Amazon but ridiculously slow to some other consumer ISPs). If you don't want to serve from home, i'm sure many entities would be happy to donate disk space and bandwidth to help a project like this, setting up a mirror list like we have for distro repositories.

Also, we may be taking the problem the wrong way around: do these multigig files need to be accessed by everyone from a web browser? No, it's a dump file used by specific people in specific circumstances. Then why are we using HTTP for this in the first place? In this case, only publishing over Bittorrent/IPFS makes sense and many people will happily seed, pushing costs toward 0 for the publisher (and very close to 0 if you only push to a first circle of mirrors who can then push to more mirrors, some of which can be declared webseeds in your torrent).

You're not the target audience.

Startups growing fast are the secondary audience.

The primary audience is large enterprises where their internal IT costs <<more>> than the cloud costs. Plus internal IT provides those resources after 6 months...

Most people that use cloud computing aren’t stuck with the bills the companies they work for are.

As to difficulty, they “solve” organizational problems by avoiding sticker shock when someone wants 100+k in equip that’s often a huge number of hoops to jump through and possibly months of delays, a giant bill every month and nobody a complains about the electric bill etc.

> Most people that use cloud computing aren’t stuck with the bills the companies they work for are.

you can rest assured that even the largest company will come looking for the person responsible for increasing an expense by that large of a percentage. So maybe it doesn't come out of your personal checking account but you will certainly pay for it.

That’s assuming it’s a large increase in the bill for no reason.

It’s easy to justify having a larger bill with more traffic. “A retail store isn’t going to complain that they need to buy more stock after selling more stuff that’s just a cost of doing business.” Meanwhile it can be hard to justify a capital expenditure just because traffic increased.

> 600GB/day is about 55Mbit/s. not really it was minimal traffic then sudden bursts of gigabytes. Of course throttling the big spikes would actually have been a good idea in hindsight to give an early warning.

> but at least it wouldn't automatically ruin your bank account whenever you got a little traffic.

This only happens when consumers fail to set budget alerts. Troy could have saved himself $10k with 15min worth of work.

Reply on news.ycombinator.com