A while back we managed a site that would serve ~700 TB/mo and paid about $2,000 for the servers in total (SQL, Web servers and caches, including traffic). At Google's $0.08/GB pricing we would've ended up with a whooping $56,000 for the traffic alone. How's that justifiable?
Outside of the Cloud Providers Traffic is dirt Cheap, Hetzner includes 30TB traffic inclusive in their dedicated server i7 Quad-Core Skylake 64GB DDR4 RAM, 2x250 GB SATA 6 Gb/s SSD for 39 euro /month:
https://www.hetzner.de/us/hosting/produkte_rootserver/ex41ss...
If you don't want to be shaped after you exceed 30TB, Hetzner charges €1.17 per additional TB, so 700TB would come to €783.90 total.
Whereas ScaleWay include unlimited traffic in their bare metal servers starting from 12 euro /month:
https://blog.scaleway.com/2016/03/08/c2-insanely-affordable-...
If you mean that you can have a load balancer in front of it managed by Amazon, that's true for dedicated as well (Akamai, CDNetworks, Limelight, even Leaseweb). Managed databases are available from most providers (usually without an API, but you can find them with an API as well).
I've had one server with them for about 3 years, and another for 2 years, and haven't run into a hardware issue yet. Obviously a hardware issue could happen at any time, so anything I can't live with being offline until I can restore from a backup is configured with redundancies, including a Digital Ocean VPS just in case the datacenter my servers are in goes offline.
From my monitoring, however, I tend to see a short network blip about every other month, but it's less than a minute at a time. All other outages I've had were my own fault.
https://robot.your-server.de/order/market/sortcol/ram/sortty...
The gotcha is the allowable bandwidth for their auction machines are lower than their normally priced servers.
https://www.hetzner.de/us/hosting/produktmatrix/rootserver
I got lucky and found a 32GB machine with 4 Samsung SSDs in their auction and it has served me well for testing. I write about it my blog below:
http://gitsense.github.io/blog/benchmarking-march-14-2016.ht...
I think I'm paying about 68 euros a month for it. The Canadian dollar has taken a beating, so it's not as good of a deal as it use to be, but it's still a good deal none the less.
$120-$230 for first TB of egress bandwidth depending on where it goes. No thanks, I can get 2 TB for < $20 elsewhere.
These bandwidth costs leave small businesses, and individuals like myself, staying with the smaller competition. I suppose their reasoning is they can chase after that single $400-600 million contract. One major client like that is worth as much as ten million of us little guys paying $50 each. The big cloud providers exist to to serve gigantic enterprises. The rest of us are a drop in bucket and not worth the effort.
Do you have any examples? It seems like it's always been a grow-and-become-profitable-or-die-fast niche.
They need two traffic prices.... Fast low latency web traffic for the current 10 cents per GB. Slower more laggy CDN type bandwidth for like 10 cents per TB.
Edit: yes. (https://cloud.google.com/compute/docs/load-balancing/http/cd...) But it's more of a CloudFlare competitor—a distributed caching reverse-proxy with a 4MB object cachability limit. Costs $0.008/GB, which is cheap compared to a real CDN, but expensive compared to CloudFlare's "free."
And those rates are still in crazy territory compared to most alternatives other than Azure and AWS which have equally messed up bandwidth pricing last I checked.
I build caching solutions for customers that want to store their data in S3 or Google Cloud Storage, because the bandwidth prices at the big cloud providers are so out of whack that as soon as someone uses lots of egress (few TB a month or more), you can often cut your bandwidth costs by 80%+ or more by getting some dedicated cache servers to put in between your users and your cloud storage. That is after the rental and management costs for those cache servers are included.
(the reason for this rather than building storage solutions is that if the above fails you don't lose data. If you trust your abilities or service provider, building a multi-location storage setup with 3+ times redundancy that beats S3 etc. on cost by a large margin is fairly straight forward... But it's often easier to sleep at night if you have other people do the risky stuff..)
Nor everybody wants to run the next Netflix or Dropbox in terms of bandwidth consumption. Even if you did, keep in mind that Netflix does not host the videos in the cloud.
What kind of site would serve that volume of traffic and not have 56k for operating expenses? I mean, I can think of a few examples like Wikipedia maybe, since they are non-commercial and such, but for a commercial business? Maybe 4chan moves that much without a lot of revenue I would think, or maybe... imgur? but not really sure, I mean, it would seem like they could get that amount easily via ads alone.
What was the use case here?
Also, I think that 56k for traffic alone kind of depends on context. I mean, how much does Netflix pay for serving their volume of traffic?
What I'm saying is, isn't 700 TB a month something that would probably be very expensive no matter the context? Just storing 700TB would cost a lot, no?
I'm really curious about your use case here.
Is that a fixed-cost sustained pipe though? I was under the impression that (at least at the backbone level) those contracts got more costly the closer to full that your pipe was.
It's obvious that I had no idea about costs. I honestly thought it would be much more expensive.
The total dataset was just about 3TB, so storing it was not an issue.
It does make sense. Thanks for satisfying my curiosity :)
I still didn't make my own "clone", because I can't afford machines to start selling it.
I just thought that moving that much data would cost some serious money. Apparently that's not really moving "that much data".
Wrong assumptions on my part :/
AWS has a lot of 'free' services, which still have to be paid for. Some of those free services are things that benefit both the client and AWS, but would be avoided by many if folks had to pay for them (like IAM credentialling)
Supposedly this has saved them immense amounts of money.
If you can load balance at the client, then you can "talk" to any server at the edge and don't need a router or proxy, so the net result is that you are only paying for whatever bandwidth comes with your OVH (or whatever) boxes. Effectively, you're buying bandwidth and the computer/storage/power/rackspace/etc. that comes with that bandwidth is free.
And yeah, it's ridiculously cheaper than AWS or Google's Cloud Platform to do things this way.
Because The Cloud(tm) IS cheaper--when you start and don't have any real bandwidth or CPU usage.
Whereas, every colocation facility I have quoted wants you to commit to a minimum of $500 for some partial cabinet. So, The Cloud(tm) wins the contract and gets to bill in increasing amounts when usage finally goes up.
Finally, how many real system administrators still exist who can provision your systems, configure the network, and understand how to connect everything to the network without getting p0wn3d? If you don't have that person, you can't escape The Cloud(tm) even if you wanted to.
... a lot? Has there been some shortage of network/infrastructure people lately?
In the end, the cloud makes sense in a lot of scenarios.
E.g. your example: Understanding how to connect everything to the network without getting hacked is far easier when your private network is physically wired to a separate switch, and your public network is physically behind a firewall and there's no configuration mistake in the world you could do that would change that, so the problem-space to get basic levels of security is reduced to configuring the firewalls correctly.
Still plenty of room to shoot yourself in the foot, but in my experience far less so than having people configure their own networking on AWS.
As or pricing, yes, if you want to do colo, the initial costs are higher. But dedicated rented servers with monthly contracts are also typically far cheaper than AWS for anything that stays up for more than ~1/3 or so of the time (obviously depends on the hosting povider). If you regularly spin up lots of instances for a short period of time, you should use AWS. But the moment you stop spinning them down again, it's time to rent capacity somewhere else.
You need IOPs? Suddenly you are paying for a premium instance type.
You want replication and/or geo-redudancy with that? Now we're talking $$$ :D
Disclosure: I work on Compute Engine.
* Note: I went to use Compute Engine's cost calculator but it appeared the site was down / under heavy load?
Just buy dedicated servers or VPSes, no datacenters or staff needed. The hosting provider takes care of the servers, staff and the datacenter.
As a business, I wouldn't do it until the cost of the bandwidth+hosting exceeded the cost of an extra, dedicated employee to manage the VPS server(s).
The hosting company takes care of the VPS servers, just like Amazon takes care of the AWS servers.
I'm not saying don't optimize, but I am saying that you shouldn't switch infrastructures unless you are saving enough to cover additional talent.
Why do you think you don't need that extra person to manage the instances in a cloud setup?
My experience is the reverse: It tends to take more man-hours per instance to manage a large cloud setup, because there are many more spinning wheels. The overall complexity is often vastly larger. In fact, I have clients I manage physical servers for where the time taken per server is on average still far lower than for cloud instances even including the 2+ hours lost on travel per visit to one of the data centres if someone has to physically go in (rather than rely on "remote hands").
This is before factoring in typically higher utilization rates for the dedicated hardware, because it's easier to customize it to get the right balance of RAM, CPU and IO for your workload. The result is usually fewer dedicated servers than you would have cloud instances.
Working is better than not working perfectly.
One theory of mine (perhaps uninformed; I'm not really a networking expert) is that because of the dynamically configurable nature of their systems, they need to use routers rather than relatively dumb and cheap switches at almost every level - in order to have flexible networking and still maintain isolation between customers.
This could get quite expensive if you have to pay Cisco/Juniper for this. If this is true Google will have quite an edge with their software defined networking here, I would guess.
http://drpeering.net/white-papers/Internet-Transit-Pricing-H...
[1] http://drpeering.net/white-papers/Internet-Transit-Pricing-H...
Nike advert?
SDN is going to turn the cost structure on its head -- I wouldn't want to be a network guy now, easily 60% of tasks are getting vaporized in the datacenter.
From my vantage point, it's going to be at least another five years before the cost structure really does turn over on it's head for folks below the hyperscale level
The ASIC is just a hardware offload for known routes. Unknown routes, admin work, and Ping packets are handled by the x86/ARM CPU. It's not too different from offloading graphics work to the ASIC on your graphics card, or your mining to your Bitcoin ASIC.
What, morally?