DigitalOcean opens a Singapore Datacenter | Hacker News Reader

DigitalOcean opens a Singapore Datacenter

digitalocean.com

108 pointsbeigeotter70 comments

Ok, this is nonsense!

DigitalOcean did not "open" a data center in Singapore. All they have done is rented some racks / colo space from Equinix in an existing facility.

http://www.equinix.com/locations/singapore-colocation/singap...

They're just another customer - to spin and imply they have built or opened a brand new data center is ridiculous.

"Our decision to open a new datacenter" is really "We've located servers in Singapore". What's wrong with telling the truth?

Just more evidence that DigitalOcean is all PR and hot air after their pot-shots at Linode and brushing off security issues as "features".

You gotta love how some people can spin investments in the thousands of dollars as if it was millions.

They haven't said anything untrue. They're open about the fact they're working with a colo provider and using a common definition for "datacenter" appropriately.

While "datacenter" can refer to the physical building, it's also commonly used to refer to a set of server/storage/network resources physically cordoned off for your use in a colo.

Using this definition isn't just common with hundreds of other hosters, its well standardized in IT circles, and used as the official definition of the word "datacenter" by numerous industry groups, regulators, analysts, and the US Federal government.

For instance: http://searchdatacenter.techtarget.com/definition/data-cente... http://www.whitehouse.gov/sites/default/files/omb/assets/ego...

The whitehouse.gov pdf above says: "under the FDCCI, a data center is now defined as a closet, room, floor, or building for the storage, management, and dissemination of data and information. Such a repository houses computer systems and associated components...<snip>... housed in leased (including by cloud providers), owned, collocated, or stand-alone facilities."

I'll respectfully disagree. They are implying creation and ownership of an entire facility. They want to market themselves as being bigger and stronger than they actually are.

I guess hustling and bending the truth is rule 101 in marketing for start-ups these days, but it doesn't have to be like that.

When Blue Bottle open a new coffee shop, that's what they call it, an opening. If other coffee shops stock their coffee, Blue Bottle don't claim to have opened a new location, they simply say their coffee is available or served there.

Their announcement uses the phrase "Working with Equinix to ensure the highest quality facility". Since Equinix is a colo provider, that implies they're running in a colo.

If you read their FAQ (https://www.digitalocean.com/faq), they're pretty clear that their other datacenters are in 3rd party colos: "We have datacenters located in the NYC area (located in the Equinix and Telx datacenters), San Francisco (Telx), and in Amsterdam (TelecityGroup)."

In case you aren't familiar with equinix, they house almost everyone's "data center". Most of AWS is in equinix along with most financial institutions. Very few groups actually physically build a DC these days.

>"Our decision to open a new datacenter" is really "We've located servers in Singapore".

Did they locate the servers in singapore? No, your 'truth' is also non-literal.

What matters to customers is that there's now a Singapore option in the datacenter drop down box.

One would hope they've located some servers there.

Otherwise they're just reselling servers, which makes their claim to "open a new datacenter" even more of a farce!

I'm not sure who started this trend, but they're certainly not the first to do it. They're just copying Cloudflare and others who use misleading terminology to appear bigger.

FWIW CloudFlare uses the same terminology. It doesn't seem to be particularly egregious.

Happy to hear about the new opening, DO. Happy customer here - thanks for providing what has proven to be solid service and support for me (far better than my previous 'budget provider').

I am seeing a lot of complaints about DO that has nothing to do with this announcement... Traceroutes and ping times from the region - legit. Debate on the claim of 'opening a data center' (vs renting rack space) - legit (at least on topic). Complaining about lack of DO features - off topic and noise...

Let's be honest about what DigitalOcean is (and they may not like my categorization - I don't know, I'm just a customer) - they are a low price / budget hosting provider... I am not saying that people's complaints about lack of features or problems aren't legitimate complaints, but that they don't particularly belong here or add value to this discussion. DO has been on here a lot - whining about wants and complaints on every post is lame...

You want to voice your complaints? Write a post about why DO doesn't live up to your dream of all that a $5/mo VPS solution should live up to on your little blog and see if it makes it to the front page - then you can rightfully whine about lack of IPv6 all you want and know that you're not just thread dumping on a young small company's enthusiastic announcement about expanding their services. Or... Here's a thought for all of you super awesome critics who clearly know what's up way more than DO does - go make something superior and sell it for $4.50/mo!!!

Perfect example from another comment: "I'm worried about DigitalOcean. I think their priority should be to try to approach feature parity with Amazon or Rackspace."

Maintaining/achieving solid quality and reliability with their current, or a slightly extended feature set, while maintaining their awesome price point is FAR more important than feature parity with the biggest VPS hosts in the world. We don't need another hosting provider that has every feature ever at a premium price. We need a provider with a smaller feature set that is more affordable.

That doesn't mean no new features, but we are certainly not rushing products out the door, market pressure or not. We know what we're doing. :)

Something seems off with the routing. I'm on one of the big 3 internet providers in Singapore, and I'm getting a 300-400ms latency to their ping server (speedtest-sgp1.digitalocean.com) and to a droplet which I just started.

    DO <-> SL ~2ms
    DO <-> AWS ~2ms
    Me <-> SL ~15ms
    Me <-> AWS ~15ms
    Me <-> DO   350ms

Check your routing, I'm on the east coast with Level3 and I get 250ms:

PING speedtest-sgp1.digitalocean.com (128.199.255.139) 56(84) bytes of data.

64 bytes from 128.199.255.139: icmp_req=1 ttl=51 time=256 ms

64 bytes from 128.199.255.139: icmp_req=2 ttl=51 time=256 ms

64 bytes from 128.199.255.139: icmp_req=3 ttl=51 time=256 ms

64 bytes from 128.199.255.139: icmp_req=4 ttl=51 time=256 ms

64 bytes from 128.199.255.139: icmp_req=5 ttl=51 time=256 ms

--- speedtest-sgp1.digitalocean.com ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 4005ms

rtt min/avg/max/mdev = 256.248/256.456/256.851/0.211 ms

Not the best from Australia yet either.

    --- speedtest-sgp1.digitalocean.com ping statistics ---
    49 packets transmitted, 45 packets received, 8.2% packet loss
    round-trip min/avg/max/stddev = 262.448/360.721/530.517/66.088 ms

Despite what DO said in their announcement, my experience says that Australian users are better served by servers in California than Singapore.

They're all terrible, but SFO does look better for me than the rest.

    --- speedtest-sgp1.digitalocean.com ping statistics ---
    50 packets transmitted, 49 packets received, 2.0% packet loss
    round-trip min/avg/max/stddev = 281.718/334.077/458.775/43.216 ms

    --- speedtest-ams2.digitalocean.com ping statistics ---
    50 packets transmitted, 48 packets received, 4.0% packet loss
    round-trip min/avg/max/stddev = 368.574/423.556/581.576/46.758 ms

    --- speedtest-nyc2.digitalocean.com ping statistics ---
    50 packets transmitted, 47 packets received, 6.0% packet loss
    round-trip min/avg/max/stddev = 283.225/338.437/428.314/31.986 ms

    --- speedtest-nyc1.digitalocean.com ping statistics ---
    50 packets transmitted, 49 packets received, 2.0% packet loss
    round-trip min/avg/max/stddev = 284.853/339.193/424.864/29.139 ms

    --- speedtest-ams1.digitalocean.com ping statistics ---
    50 packets transmitted, 47 packets received, 6.0% packet loss
    round-trip min/avg/max/stddev = 352.165/453.513/584.099/52.971 ms

    --- speedtest-sfo1.digitalocean.com ping statistics ---
    50 packets transmitted, 48 packets received, 4.0% packet loss
    round-trip min/avg/max/stddev = 205.293/324.091/445.077/54.935 ms

It's a coinflip. Some users will route straight to Asia (because their ISPs are buying the right bandwidth) and be better off with Singapore. Some will route to Asia via the western US, and will be better served from California.

It's bad enough that when Blizzard sold region-locked Starcraft 2, the Australian version of the game was the only one which gave players access to two regions - Southeast Asia and North America.

Fortunately, it's 2014 and both AWS and Rackspace are in Sydney now. If you're serving Australians, I think you should just use one of them.

tpg user from sydney, routed through SanJose, 4xx ms

  --- speedtest-sgp1.digitalocean.com ping statistics ---
  10 packets transmitted, 10 received, 0% packet loss, time 8998ms
  rtt min/avg/max/mdev = 397.643/403.366/419.058/5.814 ms

My tests from dedicated servers hosted with the big three in Singapore showed this:

    StarHub > DO ~70ms (via Tokyo, with NTT)
    SingTel > DO ~70ms (via Tokyo, also with NTT)
    M1 > DO ~210ms (via Los Angeles)

I don't have a droplet with DO to perform the reverse path.

So this looks quite unoptimised for actual Singaporean users right now.

I'm glad you pointed out the importance of checking the reverse path. Traceroute can often be misinterpreted. I'd recommend the NANOG presentation "A Practical Guide to (Correctly) Troubleshooting with Traceroute".

https://www.nanog.org/meetings/nanog45/presentations/Sunday/...

Not best from Thailand either.

This is some traceroute result from True IDC

13 ae-6.r21.tkokhk01.hk.bb.gin.ntt.net (129.250.4.26) 61.168 ms 59.223 ms 59.476 ms

14 p64-2-1-2.r22.tokyjp01.jp.bb.gin.ntt.net (129.250.4.36) 108.462 ms p64-2-1-3.r23.tokyjp01.jp.bb.gin.ntt.net (129.250.4.56) 112.855 ms p64-2-1-2.r22.tokyjp01.jp.bb.gin.ntt.net (129.250.4.36) 122.359 ms

15 as-6.r21.sngpsi02.sg.bb.gin.ntt.net (129.250.5.157) 132.448 ms as-5.r21.sngpsi02.sg.bb.gin.ntt.net (129.250.4.151) 132.776 ms as-6.r21.sngpsi02.sg.bb.gin.ntt.net (129.250.5.157) 142.057 ms

16 ae-6.r00.sngpsi02.sg.bb.gin.ntt.net (129.250.6.105) 138.498 ms 139.374 ms 139.766 ms

17 116.51.27.150 (116.51.27.150) 147.845 ms 146.281 ms 147.800 ms

18 103.253.144.242 (103.253.144.242) 136.963 ms 131.681 ms 137.451 ms

Another HNer in Thailand , I'm in Bangkok where are you ?

From Germany over France, Netherlands and the US:

--- speedtest-sgp1.digitalocean.com ping statistics --- 50 packets transmitted, 50 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 277.878/291.729/306.448/5.964 ms

Checking their BGP, they're only peering with NTT and Teliasonera: http://bgp.he.net/AS133165

Honestly, I don't understand this stuff but possibly this is why things are so bad...?

Peering matters a lot. Routing is often more of a financial decision than a technical one.

This is actually expected behavior. I'm Singaporean and we are well-known for this. Each request, even from Singapore to a Singapore-based site takes multiple hops

Peering from Sweden via Telia is excellent, considering the distance.

    Ping statistics for 128.199.255.139:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
    Minimum = 213ms, Maximum = 215ms, Average = 214ms

A response from support:

"Australia is hit or miss depending on what provider or area your in. We had some delays in our peering installs and you should see some improvement within the next couple weeks. I will keep you update every few days on the progress since your in one of those areas we cannot immediately optimize the routing too."

So sounds like those delays are the issue.

We are seeing some issues with the network transit providers that we selected in Singapore where some of the traffic is being routed optimally. Bryan has been working hard on determining the best mix of network transit providers and peering to setup to get this resolved as quickly possible, unfortunately when it comes to provisioning network cross connects with service orders and contracts its not a 24-48 hour turn around, but we are actively working on this very diligently.

I'm worried about DigitalOcean. I think their priority should be to try to approach feature parity with Amazon or Rackspace. As of now, there are many features that are missing, such as private networking (only available in NYC2), CDN, elastic IP, load balancing, etc.

How about the ability to support attaching metadata to any instance?[0] How difficult is this to implement? More importantly, how would a devops person manage many instances without this feature? Isn't ex-EC2/Rackspace/<others> the target market for DigitalOcean?

--- [0]: http://digitalocean.uservoice.com/forums/136585-digitalocean...

Not to mention IPv6. I like DigitalOcean but am starting to think of completely ditching them not just because they don't have it (delays are understandable), but because the last word from them (around August or September last year) was that they'd have a public beta in October, and then gone completely silent.

No public beta happened, and now they won't respond to any customer queries about it. I'm fine with there being delays, but very not fine about the silent treatment (or vague answers like 'it's in development' - actually give some detail of what's been going on because it's been 'in development' for something like three years now).

I guess the reason is because their focus is primarily on expansion. They're trying to put on their big boy pants too soon.

Or they're simply under investor pressure to put on those big boy pants over achieving feature parity.

Or they're simply, you know, responding to demand.

You hit the nail on the head. We provided, and more specifically, I, directly did, estimates on when we launch certain features.

Unfortunately those deadlines have come and gone. That was because those estimates were rooted in timeframes around how we were able to write code in 2012 when we didn't hit the large growth spurt we did in 2013.

As a result the majority of 2013 was about meeting customer demand. Luckily we've finished out the year very well in scaling the various teams within the company and adding in the necessary layers of management so that we finally feel that we are getting back on track with engineering again.

With the launch of Singapore there are actually alot of updates under the hood that customers don't see but the hypverisors in Singapore are running on a completely new version of our cloud backend.

As we close out February we are reviewing our product roadmap and will start providing feedback again on estimates and hopefully be once again much more accurate in those estimates.

One of the other items we've considered is putting together a blog post about our 2013 in review and specifically focusing on what happened to our product roadmap and what delays we hit and why, and how we are looking to get past those challenges in 2014.

Thanks!

And still no IPv6 support :(. When will they make it a priority?

First thing I said when I came in - glad to see others say the same!

I just figured out I can run IPV6 hosts at home on Comcast and ping them from an IPV6 ping gateway page. I'm still scratching my head on how that works and whether or not my router allows all IPV6 packets inbound to all hosts. Tried to test on AWS, but that's not working. Went to my DO account, and still no support.

Rackspace supports IPV6, but I think something wonky happened to my account when I was testing the developer credits for them. I need to get it fixed.

Anyone have IPV6 at home and want to help me test it?

Sure.. got a bunch of VMs&machines I can test from.

But first order you could just traceroute with Sixxs[0]. That way you will at least know if the icmp6 packets reach their target unharmed. Your router might still be blocking all incoming connections per default (which is probably a good idea for home networks).

[0]https://www.sixxs.net/tools/traceroute/

Sure; drop me a mail and I can test from the UK if that's useful.

(Miredo tunnel here at home, or native IPv6 connectivity on my vps provider - Bytemark, in Manchester, UK.)

Though I'm just about to sleep so it'll be ten hours or so ;)

Thanks for the offer. Got Rackspace going again and tested. Wide open. O_o

Depending on what your router is running (hopefully OpenWRT), ping6/ICMPv6 should be allowed to any host by default. Check out your firewall rules and see what it says there. ICMPv6 is an important part of IPv6, and in general not being able to ping something isn't really a security gain, so much as a usability loss.

On my OpenWRT setup, the default is to rate limit ICMPv6 to 1000 requests/second and to limit the response types.

I'm able to SSH into my internal servers. Configuring OpenStack now to provision the addresses for instances as well.

I have an Asus RT-AC66U. Tried to flash with OpenWRT and a few others and failed a few months ago. It would appear it DOES NOT have a firewall enabled, so I'm wide open, so to speak.

Ooh, that's not good. I would suggest you put up a firewall directly on as many devices as you can. Then, try to get a router upgrade, or a router that can handle an IPv6 firewall. I think I know the router you have and I believe TomatoUSB can run on it, which has ip6tables installed.

I know it seems like it's not good, but I was thinking about it and even with Comcast's /64 aggregate, I have a billion BILLION addresses available inside my network. If you could scan at a billion addresses a second, it would take 30 years to scan all of Comcast's IPV6 addresses. That's crazy.

Well, not really. The first 64 bits are not all possible. They are subdivided, since some addresses are link-local, some are multicast, etc. Then, Comcast only has a certain allocation of that. On top of that, could one find a patter in how they allocate their addresses?

The second 64 bits are also not quite random. Most of your devices will autoconfigure using radvd. This means that the second 64 bits depend on their MAC address. Now, if I knew of an exploit to, say, a printer or a NAS device, I would know the MAC address range. My guess is that I could probably reduce the 128 bit address space to something like 100 or even 90 bits.

Second, and this makes it all the above a moot point, don't your devices connect to the internet? Any time they connect to a site, that site knows the IP address and that data may be used either explicitly or leaked and used by someone else. Everyone between you and the site also knows the address.

Lastly, if you ever set up a DNS record for any of these addresses, they are then visible to others even with some scanning if you don't ever publish the actual names.

Long story short, there is hoping you don't get hacked and there is knowing you have a firewall that only allows what you want in.

Yeah, I think this is an issue with the router (I have it too). I was using a tunnel and had to turn it off because you can't get it to go through the firewall.

I think you can fix it if you telnet into it and manually set up iptables properly, but it overwrites the configuration on update.

Pretty poor form that ASUS haven't fixed that yet... It's annoying because otherwise it's a very nice router.

Called Rackspace support. Turns out I owed them $7 due to a bad card number. Signed back up with the developer's discount! I really appreciate the fact Rackspace has added support for IPV6.

I wasn't aware of this, and perhaps it's a 'feature', but all my boxes in my house are wide open to the Internet over it. Investigating...

How does DigitalOcean compare to RamNode? I've been using a small VPS at RamNode and it's been great. It offers pretty much the same features (SSD, 1 CPU, 1 Gbps connection) as DigitalOcean (plus RamNode offers 16 IPv6 addresses), yet it's half the price. The only downside to RamNode is the 500 GB of bandwidth compared to 1 TB at DigitalOcean. Maybe I'm missing something, but why would one use DigitalOcean over RamNode for VPS hosting?

500GB bandwidth you say? That is the plan with only 128MB of memory; you can't run many things with so little memory. The 512MB plan at RamNode is $7.5, 50% more.

Is that factoring in the coupons they keep putting out there? I paid $100.44 for a year after their 38% coupon (38RLY) and signed up for 1024 MB RAM plan, 150 GB disk, 2 IPv4 addresses, 3000 GB bandwidth. It's great.

I only considered the base monthly price; you can definitely get it cheap with coupons and yearly plans.

If you're looking for a bargain, I found this:

    2GB RAM, 50GB disk, 2TB bw, 2 IPv4, 100Mbps
    STEALOFADEAL (coupon) - 12 months, $40
    Chicago, IL - https://billing.chicagovps.net/cart.php?a=add&pid=160
    Buffalo, NY - https://billing.chicagovps.net/cart.php?a=add&pid=161
    Los Angeles, CA - https://billing.chicagovps.net/cart.php?a=add&pid=162
    Atlanta, GA - https://billing.chicagovps.net/cart.php?a=add&pid=163a=add&pid=163

ChicagoVPS are terrible. If you Google ChicagoVPS the suggest searches are "ChicagoVPS down" and "ChicagoVPS hacked".

There's 10 million threads on Lowendtalk bitching about ChicagoVPS. Here's one from last year. http://lowendtalk.com/discussion/7668/chicagovps-has-lost-th...

You get what you pay for.

I've sworn off cheap VPSes - most are running WHM as their control panel/billing and they're a target for DDoSes and attacks by skiddies.

I have too. I used some cheap vms for about 9 months, but after the host lost vms including backups (I had my own backups, but I still had to set everything up again), and relatively frequent outages, I switched to Digital Ocean, and couldn't be happier, and the price is pretty good. I figure I'll switch to Linode if I start having problems, but it's been great so far (knock on wood).

Why are those complaining about routing issues posting ping outputs instead of traceroutes?

It doesn't matter. Even if they posted traceroutes, they'd only be from one direction which is near useless anyways.

I started a SG droplet about 5 hours ago. It's still down. They're suffering some outage and it's a shame it isn't mentioned on their status page. #fail

https://twitter.com/kaihendry/status/433451127657357312

I'll stick with GPLhost for now http://webconverger.org/servers/

They need to keep on top of existing demand as well and start running over-capacity. Not joking but it's pretty long wait to get an instance up in a preferred DC sometimes which is putting a lot of us off the platform.

I am going to keep machines at Digital Ocean, but they have had the most failures/downtime I have seen from any host that I've used in nearly a decade, including ec2, rackspace, and gogrid (if there is anyone left using gogrid.)

One problem we've run in to is an instance fails, but it can't be brought back up because the backup image is only located in the datacenter that is already at capacity.

A million upvotes if I could.

I didn't actually consider that last point. That has effectively killed DO for me instantly from a risk point of view and I'm going to immediately migrate my two machines off it. Literally it's 22:50 here tonight and I don't think I'll sleep until it's done.

Have some company credit left over with Azure so will throw it on there on Ubuntu for now.

It used to be that it was possible to take a snapshot of a DO droplet without shutting down the droplet, provided the droplet had 1GB of ram or higher.

Does anyone know if this is still the case?

I don't know if that was ever the case, but at this point you do have to power off the droplet to make a snapshot: https://www.digitalocean.com/community/articles/digitalocean...

You can take a "backup" without powering down, but not a "snapshot".

I hope a Canadian Datacenter (preferably westcoast) is in the cards soon!

Make a uservoice post about it if there isn't one.

I wish they would add IPV6 support.

Reply on news.ycombinator.com