Digital Ocean: Limited Public IPv6 Beta Release | Hacker News Reader

Digital Ocean: Limited Public IPv6 Beta Release

assets.digitalocean.com

71 pointsjgillich59 comments

Now, how long has this been "coming soon" with absolutely no word whatsoever?

https://digitalocean.uservoice.com/forums/136585-digitalocea...

Oh, only nine months having an entire customer base entirely forgotten. This does not good support make.

DO has a history of doing this; see Allow Custom Images [0], with over a year of no comment, and Give option to use the Droplet's own bootloader [1], with 14 months of no comments; not even a "we're working on this". The latter shows DO's utter lack of technical competence; if they cannot even properly prepare a system image [2] or actually use their virtualization platform, what the hell can they do? Apparently, only marketing.

[0] https://digitalocean.uservoice.com/forums/136585-digitalocea...

[1] https://digitalocean.uservoice.com/forums/136585-digitalocea...

[2] https://missingm.co/2013/07/identical-droplets-in-the-digita...

My ISP has had IPv6 "coming very soon" since 2009.

Apparently they've had a "Custom Image" beta in the admin UI for over two years [0]. There's also a CoreOS image request [1] that's been sitting around for almost a year. They just entered beta, and there's not really a good way to use them on DO without abusing Fedora, apparently.

[0] http://digitalocean.uservoice.com/forums/136585-digitalocean... [1] https://twitter.com/digitalocean/status/176338623053045760

That's a great point and we totally missed our roadmap in 2013, by about 9-12 months. This is something that Zach Holman at Github addresses in why Github doesn't really communicate about their roadmap:

https://github.com/holman/feedback/issues/534

In our case it was because we were basing all of our roadmap on how we operated in 2012 when we were a much smaller company, think 5 people, servicing a much smaller customer base, say 500 customers.

It was early 2013 that our growth really started and we tried to remain optimistic about our ability to launch features. This turned out to be completely unrealistic. We had to scale our customer support department which at that time was basically myself and Etel who is our community director from 0 full-time people to 17 today. Our engineering team was all cofounders which we have since scaled to 10, and we had no SRE team which we've since also built out and recently hired Mark Imbriaco from Github as our VP of Ops.

Our tremendous growth was very exciting but it stressed every single area of the business. We had to raise two rounds in 2013. Though we closed our Series A in 2014 we actually began that process in early November of 2013.

As the customer base grew and began we also started to encounter more scale issues which began to shift our engineering focus away from feature and product development to bug fixes, refactoring, and re-engineering.

We would have wished that we could move faster and pay down technical debt so that we could move into developing features sooner but given our tremendous growth last year of scaling from 500-1,000 customers at the start of the year to close to 100,000 by the end of the year it simply wasn't possible.

Along the way we completely broke a few promises about when we would release new updates and features. Now we are finally at the point that we feel confident that we are able to develop and roll out new features and we have begun doing so.

We've certainly learned some valuable lessons along the way such as being very careful about the public promises that we make. It's hard to say at the moment, but I don't think that we will follow Github's model of being secretive with our roadmap, but we will certainly be more cautious with our estimates given how we have seen so many different factors play into our ability to release features.

I think any startup that has scaled has gone through similar problems and so it's a story which I'm sure has been repeated many times in history. We could have done a better job of staying ahead of these changes and more pro-actively communicating that we would be falling behind our product roadmap as a result and that's a personal failure on my end as I lead product in the organization.

Much like a startup evolves, matures, and changes overtime, I too have had to go through a similar process of learning and re-learning my role both as a cofounder and also as a product lead.

It's been very dynamic and interesting and certainly has been a tremendous growing experience and I've finally been catching up with the rest of my work now that we've settled so many different problem areas in just scaling a company and a business.

One of the first things was to get back on top of UserVoice and begin to provide more updates and visibility into the things that we've been working on and what's coming up. For other items we are still in the process of hashing out the details and providing better estimates so in those cases we are holding off before we make another public statement so that we can ensure we set the correct expectations.

Believe me that there is nothing more troubling to us as a company than letting down our customers, it is never our intention, and it is never out of ill-will, often times it happens out of being too optimistic. Like any trait there is a negative and positive aspect to it. As a cofounder you need that boundless optimism because there will be hard times. We had hard times when we spent 18 months with zero to little traction, and I'm sure we'll need that optimism in the future as well when we hit hard times or when the landscape shifts. During these periods it is essential. However, like anything else in life there is a balance. And, as essential as this optimism is to a startup it can also lead to issues such as these. In cases like that its a matter of being open and honest and saying yes we messed up, here is why, and here is how we are thinking about this problem in the future.

Hopefully that sheds some clarity on this area, especially since I was the one who was personally responsible for this in our company.

Thanks, Moisey Cofounder DigitalOcean

  > It's hard to say at the moment, but I don't think that 
  > we will follow Github's model of being secretive with 
  > our roadmap, but we will certainly be more cautious 
  > with our estimates given how we have seen so many 
  > different factors play into our ability to release 
  > features.

Unless your customers have a really long budgeting process, or otherwise need time to prepare for coming features, don't bother publicizing your road map. In the best case you'll only "meet expectations", and in the worst you'll disappoint people when you're late.

Hi Moisey

Any updates on BSD support?

It was started over 1 year ago without update [1] and is the 2nd most requested feature of all time [2]

(Great service by the way)

[1] https://digitalocean.uservoice.com/forums/136585-digitalocea...

[2] https://digitalocean.uservoice.com/forums/136585-digitalocea...

Edit: fix typo

Freebird! ... I mean FreeBSD!

This sort of thing is a big problem with IPv6 adoption. There's a lot of churn in internet software, server software, and hosting providers, and often many of the new guys delay IPv6 support in favor of other features. That makes it very possible for IPv6 adoption to stall or even reverse depending on the success of different hosting providers and software. Imagine a worst case scenario of a next generation web server which becomes hugely popular but which doesn't support IPv6. Fortunately that's an unlikely scenario, but unfortunately it can't be entirely ruled out. IPv6 is still a "nice to have" feature, and has been since it's introduction, that's always been it's major weakness.

Now if only they'd reconsider their decision not to deprecate Arch Linux images. I'm moving my stuff over to Linode because of it.

I run an Arch droplet and this is the first I've heard of this. Certainly makes me want to take my business elsewhere. Suddenly they don't have any bleeding edge distros.

Linode just seems to care more about their customers.

Well... If you really want, you could run Debian Sid or Fedora Rawhide, by creating a droplet form their stable version and upgrading.

It is still no excuse for not having a proper bleeding edge distro.

What policy does "grandfathered customers" refer to?

I saw that and wondered what it meant... hopefully they aren't going to try and charge customers for IPv6 addresses. Most hosts give those out for free.

There aren't any charges for IPv6.

Thanks, Moisey Cofounder DigitalOcean

Imagine that, being able to sell each addy in a ::/32 for a cent each. You'd practically be the entire economy at that point.

Grandfathered users are our early adopter customers.

When we do large rollouts we usually release new features to them first in a limited public beta test so we can collect feedback. They have been extremely helpful in helping us to find and resolve bugs before we move a feature to general release.

Thanks, Moisey Cofounder DigitalOcean

In my opinion you need to use a better term than grandfathered. I think you have a different understanding of the term than the general population. Grandfathered is not used to distinguish early adopter, developer partner, platinum/double diamond customer loyalty card, etc from the general customer pool. As witnessed in the comment below grandfathered users are the population of users that do not get screwed by a new policy.

I think you should take a look at the wikipedia page for grandfather clauses and reassess if you think this is a word you want to associate with a new feature: https://en.wikipedia.org/wiki/Grandfather_clause

Thanks, dfc. English speaking potential customer

Initially they had no bandwidth charges. Upon introducing them, the customer base was "grandfathered" in to free bandwidth. I assume this is the set of customers they are referring to, so it doesn't kind of make sense...but only if familiar with the history of DO.

Here is the deal: You cannot "grandfather" a newly created privilege to a select population--a population that never received the privilege before--if for no other reason than the privilege did not exist in the past. The grandfather bit is not only a connection to the origin of the term but also a hint about proper usage: people in the past (grandma could not vote) must have been able to and did enjoy the privilege at some point in the past.

I did not want to come off like a strict scrutiny word maven (synonym: dickhead) when I wrote my first comment because it would obscure the larger point: "grandfathered clauses" have negative connotations for new/potential customers. My goal was to be helpful and not to strut my lexicographic tail feathers.

pedantic might be a better synonym. ;)

Grandfathered is an internal term that, to be honest, we didn't expect to see on hackernews, grandfathered for us refers to customers that will never be billed for bandwidth as when we started we offered unlimited bandwidth plan. Users who are grandfathered just typically tend to be our beta group, as they well know. :)

John, Chief Technology Evangelist, DigitalOcean.

Any details on exactly what you get with Digital Ocean IPv6? The page doesn't really go into much detail - and the implementation can make or break things.

Is this a single static IPv6 address with a /64 (or /56 or /48) subnet routed to it? Or is it something else?

We are rolling out IPv6 initially with a single IP address to see how the network side of things handles that since we have already a large deployment of IPv4 and then evaluate from there.

Thanks, Moisey Cofounder DigitalOcean

A _single_ v6 address? Are you serious? I guess we'll never get rid of NAT and other idiocy if people seriously allocate single IPv6 addresses to customers ...

I'd eat my hat if it stays that way, as far as I know it's just while we test ipv6/deploy it to more regions, eventually we'll get with the program, so to speak. :D

"initially". Take a deep breath.

A single IPv6 address suggests strongly that they are doing it completely wrong. A /64 is the equivalent of a single ipv4 address, and you should never be allocated less than this.

Linode did the exact same thing with their IPv6 rollout.

Well, and unless I am mistaken, they still route the larger subnets to your interface rather than to your machine? There it nothing wrong with testing stuff with "small" subnets, even if it may be a bit pointless, but my point is that a setup that provides you with a single address just has a structure that cannot really easily be changed in a sane way into a setup that provides you with larger subnets.

A sane setup should use a transfer net on your interface, and should route your actual address space to your gateway address on that transfer net - otherwise you'll need ugly hacks such as proxy-NDP in order to be able to route addresses for sub-allocations. This flat setup without a transfer net was invented for IPv4 when IPv4 addresses became scarce, so it was a necessary evil, but it's a stupid setup that causes nothing but pain when you have billions of /64s available.

That is the reason why I don't understand why anyone would even test with single-address allocations.

Is this a single IPv6 address sharing the same /64 with stateless auto-configuration, or static?

The implementation is important, over the years I've had annoying experiences with certain providers trying to do IPv6 - but not getting things quite right:

Hetzner - Gave customers a /64, but using additional IPv6 addresses required setting up proxy NDP - which at the time was annoying as only more recent Linux kernels supported it. I believe they've improved on this since though.

OVH - Requires setting up a default route outside of your normal netmask - this isn't fun. We don't do this with IPv4 - so why do some providers do it with IPv6?

Linode - Enables IPv6 auto-configuration, so many servers share the same /64 by default, and they give accounts "pools" of /116 allocations. Sites like Google considers all of them to be part of the same /64 network - so your server will probably have trouble accessing certain resources if another server is doing excessive queries.

Only tangentially related, but Linode gives each server a /128 by default, and gives out /56, /64, and /116 for free on request. I use a /56 to do my own subnetting and other fanciness.

Sorry, forgot to mention that Linode does give more options now.

That Linode setup sounds dangerous - do they protect against NDP spoofing somehow?

Hetzner is just a normal /64 now.

Received this email earlier, though I don't have any droplets running in their Singapore center. Wish I could test this out, but I usually keep my servers in the US.

Can somebody who has a "grandfathered" account shed light on what makes an account "grandfathered" in this sense, and also what size allocations they're giving out?

(I'm aware what grandfathering is in a general sense, but unless they mean "we're releasing IPv6 as a beta to all users who were users before we made this announcement", it doesn't seem to fit here. And that would be strange)

If you look below, someone from Digital Ocean responded. It's a specific set of early adopter accounts.

Answered below, grandfathered users are our early adopters and we release features to them in a limited public beta test first to collect feedback before a general release.

There is no correlation to size allocation, it's just that they are first to be able to assign IPv6 to droplets in the Singapore region.

Thanks, Moisey Cofounder DigitalOcean

Thanks. I didn't think the type of account was linked to the allocation size, sorry for not being clear. I was just curious what allocation sizes were being given out at all.

Edit: and I see that in the interim that's been answered. Looking forward to seeing larger allocations appear as the beta progresses :)

Fingers crossed custom kernels are next.

at least 6 months out, a little bird told me.

What do you mean?! You can't have a custom kernel on a DO VPS?

Correct, there's no current official support. You can kexec to your own kernel which is kind of ok as a workaround.

Singapore is running on a new backend code base which supports this we are working on a migration pathway for the rest of the cloud so it is in the works. But as I mentioned in another area of this thread regarding product roadmap announcements we are working on being more realistic with our timelines so that we do not upset our customers. Just wanted to share that we are working on this issue and making progress, really depends on the migration and how that goes to see when we can get this rolled out.

Thanks, Moisey Cofounder DigitalOcean

General IPv6 question...is this a case where there is a benefit in being on of the early adopters? i.e. are there specific "good" addresses. Kinda like money.com is better than asoef.com.

If the subnets are allocated incrementally, you could get a shorter prefix length.

No. The only benefit is that ipv6 is useful as you get addresses.

It seems to me bit crazy that a hosting provider launched in 2011 did not have full IPv6 support on day 0, especially when the provider in question was not some shoe-string garage operation.

Not really, the issue of deployment hasn't been lack of hardware support but lack of knowledge on the part of architects and engineers. A greenfield site has enough issues getting up and running without doubling the IP stack as well.

What's really surprising is the lack of mature service provider's who are not performing incremental roll outs.

Speedtest: http://ipv6.speedtest-sgp1.digitalocean.com/

Of course you need to be on IPv6 to reach that page :)

Is my v6 so slow or is it their site? I'm getting about 300-400KB/s (unstable).

Edit: Nevermind, their v4 goes at 100KB/s.

assuming that you're in .nl, I feel like 400KB/s from on the other side of the world is not terrible.

I'm in Colorado, and I am getting a solid 10 Mbps down, 1.5 Mbps up (the up is actually what I pay for from Comcast, so it may be faster still). Not bad.

And have flash. I have ipv6 but no intention of running flash...

I am actually interested in IPv6 - will you release 1 IP per Droplet? Do they cost similar to IPv4 for hosting company, they are cheaper or are they free?

I'd also like to know that, especially as there are lots and lots of hosters out there who don't grasp IPv6 and are already breaking it before it has really taken off by creating pointless bureaucratic overhead and thus incentive to create bad network setups that will hurt us all.

As for the cost for the hosting company: No, one major reason for IPv6 is that it has a huge address space with 128 instead of v4's 32 bit addresses, so they are a lot cheaper - as it should be, there is no point in artifically making addresses scarce. RIPE members, for example, are billed for an IPv6 /32 (about 4 billion /64 subnets) like for an IPv4 /21 (2048 addresses).

In short: You should generally avoid any hosting company that allocates anything smaller than a /56, they probably haven't understood the internet.

Can someone explain the meaning and value of IPv6 (and the hurdles for using it)?

http://en.wikipedia.org/wiki/IPv6

Yes. I read the wiki page, but I was asking for a more conversational explanation. I really don't see why I was downvoted for asking a question.

Reply on news.ycombinator.com