Joyent: 54+ hours of downtime | Hacker News Reader

Joyent: 54+ hours of downtime

190 pointsenduser120 comments

I am a customer of Joyent.

For the past two days, the server on which I host some customer websites and mail servers has been down. I did not receive any warning or explanation by email or telephone. When I filed a support ticket, I was given a link to their help site with some status updates.

Status updates (paraphrased):

2011-02-26 09:22:19 GMT: server offline

2011-02-26 14:38:38 GMT: no ETA

2011-02-26 17:09:42 GMT: estimated downtime 12+ hours

2011-02-27 22:23:08 GMT: estimated downtime 20+ hours

That final estimate of 20+ hours is in addition to what is now 37 hours of downtime. Joyent has offered no reparation or backup service (e.g. backup MX service to keep mail from bouncing).

For your consideration when choosing hosting providers.

View on HN

I get the frustration. I really do and I apologize for any bump like this. I'm the founder of those old textdrive "share hosting" servers (they really have nothing to do with Joyent's current product line). These servers are ~6 years old and they're largely populated by "lifetime" customers who have tricked out various bits of their accounts enough that an automated migration has been basically impossible. We've been telling everyone for 3 years to really get off of them, we've been offering free upgrades for 4 years, including you btw, you got a free upgrade back in Nov 2007, and there's even support tickets detailing it (which you've participated on). So now we're piecing together everything that was left but those good ole FreeBSD UFS file systems take awhile.

Offering a transition from FreeBSD to Solaris that requires the effort of the customer is not free. Asking a customer who knows FreeBSD to take on administration of a Solaris virtual server for your convenience is not reasonable.

You are offering an unacceptable level of service to customers who have supported and recommended you from your early days. You were paid in advance for a lifetime of service. I took you on your word and offered lifetime hosting to friends with small websites for small businesses. If you can't make good on your agreements, I will eat the cost of moving these sites to Rackspace Cloud. I can't risk another 54 hours of downtime.

With proper backups and a fresh server it should not take 54 hours to get back online.

To even call this "Joyent" is misleading.

The packing system is the same, it's BSD userland. The paths are the same. Everything is the same. The kernel shouldn't matter. And if you had told support that you wanted us to just migrate you because nothing was out of the ordinary, what would they have done? They would have migrated you ....

And this has been quite a good deal for you. The logic of moving from something that costs you nothing to something that does costs you something is interesting.

I don't think that there's anything weird or "interesting" about the logic of moving from unreliable hosting to something more costly but more reliable. (Also, in the context of hosting service, I don't really think it's accurate to say it "costs you nothing" if it suffers a multi-day outage while you have customer servers hosted on it...)

>And this has been quite a good deal for you. The logic of moving from something that costs you nothing to something that does costs you something is interesting.

Downtime costs him. So much so that he's willing to pay for uptime.

Also, it didn't cost him "nothing". He paid for service. It wasn't free.

Your reply sounds like you don't value him as a customer anymore because he's not giving you money anymore. Nobody forced textdrive into offering the "lifetime" package. You should treat him like a valued old customer, rather than calling his logic "interesting".

> Your reply sounds like you don't value him as a customer anymore because he's not giving you money anymore.

I think you mean: replies.

Jason - I don't know you, but you're making a big mistake by constantly referring to his service as 'free'. It isn't free, it's paid for. From a customer perspective, you should consider his account as 'paid in full'.

I completely understand that he's 'getting a great deal', and that he isn't currently paying you anything, and that he may well have lasted as a customer longer than you expected, but your company offered lifetime service for a fixed cost, and he bought it. As I understand it, this was to bootstrap your later efforts.

To constantly demean his position is as insulting to him as it is to your integrity. While you may feel like you're in the right here, I can guarantee that you've lost at least one potential customer, because I will not be using Joyent. I'm guessing that running potential customers away isn't your intent here so please, don't do yourself any further harm.

>To even call this "Joyent" is misleading.

Wait, what? Who owns and operates the servers?

Joyent are the ones that have been processing my credit card every month for the last few years. What a fantastic deal this is, I get to pay $15/month and you get to laugh and comment on forums as thou this is a minor hiccup. I've built data centers faster than youve restored a server. sIX days and counting and you write as though I have been freeloading. Wonder if the English word "wanker" is known to you.

For example, read the comments below from quintinsykes and chuckmcknight.

Seems like the lesson for you here is: never give free hosting, people feel terribly entitled once they have it and their negative feedback may cause trouble in your actual products.

The hosting is not free. It was paid for in advance.

Paid in advance is still free on a month-to-month basis. That's what I meant.

Paying for a service forever in advance is a devious mis-alignment of incentives. The customer will feel that they are owed everything. The business of it won't work out.

As I said, the lesson is, don't give shit for free - even for an initial payment. Ongoing services need to be paid for in an ongoing manner.

I paid $400 for lifetime service via the vc2 drive sometime in mid 2005. I got about 2.5 years of usable service before I completely abandoned textdrive. An effective rate of $160/yr for really marginal shared hosting.

I currently pay ~$215/year for a linode account.

I will continue to pay for linode.

oh dude, I have one of those lifetime accounts. Last summer, I looked into how to migrate, but I got lost in outdated pages. Can you drop some links in this thread?

Maybe you can shut down jervis.textdrive.com if I migrate my account :)

Yes, you can contact support at help.joyent.com and they'll move your old textdrive account from jervis.

I've got a lifetime account on davie.textdrive.com and I've been waiting for my "golden ticket" for a couple years now. I never got any decent communication from joyent and everything still works, so I was disinclined to make trouble for myself. I'd be happy to migrate if it's not a lot of work.

It will take them at six days to do it and you won't have any domain or email in that time, but sure trust yourself to a joyent experience

Not till they move mine too, but this thread has reminded me to get the migration done.

Is the personal account information for the submitting individual really necessary for your point? Did the submitter give you permission to tell us about his ticket history and account actions?

Although relatively harmless, it seems like planting the seed to a disregard for privacy.

He didn't ask. My past interaction with them is irrelevant, because I only commented on their tickets enough to determine that it would take hours of my time and recurring hours of administration to take them up on their "free" offer. I decided to stay with the service I had paid for in advance.

Agreed. He really crossed a line on that one. Standard operating procedure should be to NEVER reveal ANY information about a customer, even whether or not they are customers.

One more reason to shy away from Joyent.

Edit - Jed from Linode?! Hey! You sir are my favourite support person to reach, yet I rarely get you anymore.

What I don't understand is that if the servers are 6 years old, why not migrate the old bad hardware to exactly the same OS, as virtual machines running under the virtualization platform of your choice?

A 6 year old server would have a max RAM capacity of say 32GB, you can get a newish 64GB RAM system for what, $3k or so?

In such a case, migration is basically a few shell scripts plus rsync from old hardware to new VM. You rsync a couple of times until the rsync takes very little time, then turn off services; do a final rsync; turn on services on the new VM, swap IP addresses and test.

You can even run FreeBSD/Xen on top of Solaris XVM, right?

"lifetime" textdrive customer here. I don't ever remember getting a "free" upgrade. I was just offered a $45/month upgrade for a joyent smartmachine, but while a large discount, is not 'free'.

I would be interested in getting my account into a virtual env but I can't really justify paying $45/month at this moment...what are my options?

Free upgrade sounds fine - when were these offered, and how would a person take advantage? I have one of the aforementioned permanent accounts (on jervis, even) and if migrating to a new server means that my site is less likely to fall over, I certainly would do it. However, I just dug through all the emails I ever received from Joyent or Textdrive, and none of them mentioned migrating to another server for free. Lots of stuff about how I could pay more to have more, but since things are basically fine for me (running http://imprompt.us and http://tracyfood.com both of which are low-bandwidth vanilla wordpress sites), I ignored all of that.

tl;dr: how can I go about moving my stuff? To where? And why am I in this handbasket?

I am not a lifetime customer, I pay $15/month. I have never received a migration offer and I don't have a "tricked out" account. I only really care about my email and I don't have any now for four days. Fool that I am my registrar has the email of record being the one hosted by joyent so I cant even move until you get my email up and running. I don't think you can even begin to feel my frustration. Despite the new status last night, saying everything is back up for most people, I am not. The Timeliness of support responses is also a joke, my companies email has been down for four days and you appear to be working 9 to 5 on the issue.

Frankly, I'm surprised you are still on there. I too was an original VC and I left years ago due to the support failings.

Look, we were investors in the truest sense. TextDrive was this revolutionary new shared hosting platform with a lot of promise (flexibility, no overselling, etc). Problem was that shared hosting is a commodity with no profit margins. Since TextDrive was started by people who want to build cool shit, rather than people who knew anything about operations in a customer service industry, the result is a massive series of pivots that left us out in the cold.

I'm not making excuses for anyone here, personally I think Jason Hoffman spent way too much time in the forum sucking up the love, and not enough time figuring out how to run a support operation. So he burnt his bridge with me as a customer. I won't touch Joyent with a ten foot pole no matter how good it is.

But here's the thing: that was their bridge to burn.

It doesn't matter what 200 early customers think, because those 200 one-time payments were not going to build a successful company. The bottom line is that the market for a "premium" shared hosting service got squeezed by commodity hosting on one side, and VPSes on the other. Even if they had managed reasonable uptime (which they didn't because they totally underestimated the technical cost of giving people the freedom they did), it still would have been a doomed market, because VPSes got so cheap anyway. We can complain about what a bad deal we got, but it doesn't change the fact that the last salary we paid was in 2005, and the hosting world has moved on. They did what they had to.

I generally like Joyent, and use them. So I offer this tip with a tinge of sadness:

It looks like these message are from a problem with their Howe server. That is one of their old servers, which came over from the Textdrive merger years ago. I used to have some sites hosted on that same server, and was likewise not very happy with things. It appears to me that Joyent as a company doesn't care very much about that legacy equipment and line of business.

The way to make the pain stop, is to stop using Joyent's old TextDrive servers. You can do that by leaving Joyent entirely, or by moving to their newer stuff. Either way it's a fair amount of work, depending on the complexity of what you are hosting.

I took the path of moving to their newer stuff, and have been very happy with Joyent ever since.

I had a lifetime account with TextDrive. I don't know if it was official policy at Joyent to neglect the old customers, but I certainly felt like I was being pressured to repurchase services I had already paid for. After several outages, data loss and extremely slow responses from tech support I switched to linode. No problems since.

+1 for Linode. I've literally never had a problem. I'm a huge fan and I'm picky as hell.

Not comparable for this.

Agreed. They're better.

I had a server go down twice due to outages at their Fremont data centre, but otherwise I've been very pleased.

I'm also a massive Linode fan, but here's my monthly HE-Fremont Data Center rant... don't run anything out of there (regardless of whether it is through Linode or a 3rd party).

The overal quality of their (Hurricane Electric, not Linode) operation is poor, historically they have been lax on shady people colo'ing rack space in Fremont (google hurricane electric mccolo), HE-1 is full and so little room for expansion and its on a fault line.

I've NEVER had any of these problems in Newark (which is Linode's geographically local DC as they are in NJ) or London.

So I've learnt since the first time things went kerflooie.

However I chose Fremont because it's the closest datacentre they have to Australia. Most of my readers are here.

Are the ping times and connectivity noticiably different for you between Linode's Fremont DC vs Linode's Dallas DC?

(see http://www.linode.com/speedtest/ for pingable hostnames of servers in each DC + speedtests)

If you do want to move, you can put in a ticket and they can move you between DC's with no downtime I believe or just replicate your instance and switch IP's.

It adds ~50ms to move to Dallas. That might be worth it, actually, it's not as though I'm hosting an HFT platform.

For those of us in AU, the Fremont DC is the best choice for us Linode customers.

However, I've learnt, and hopefully have others, that having everything in one basket isn't the best option, even with how excellent Linode are, they are at the mercy of HE etc.

So now we have a split between Fremont, Dallas & Newark.

It's worked pretty well, we are most happy.

Anyone running a large app or site would do well to run multiple systems with Linode across multiple sites where possible.

It's just a pity Australian bandwidth is so ruinous, or I'd move my servers here.

I was hoping to not see a comment like this. I, too, bought the lifetime account back when I was active with Textpattern CMS and joined in as part of a drive they had to raise funds (managed by Dean Allen, the originator of Textdrive and then at Joyent, who has seemingly disappeared into the ether since then). At the time I knew it was a bit of a risk but I had my fingers crossed that their word would be good. So far, my sites (on the Pendrell server) have been doing fine but I always worry about it... thus my fear at seeing a comment like yours.

Hi. This is Jason Hoffman, Dean's co-founder of Textdrive and current founder at Joyent. If you're worried about your sites on the old Pendrell, then go over to support and take the upgrade that's been offered and migrate off.

I think maybe you should resort to an approach other than one can be basically summed up as "upgrade and/or shut up." I love node.js and by extension Joyent so please don't take this the wrong way.

Also I think I have learned from this thread that 'lifetime' service packages suck because after the company closes that deal they have absolutely no incentive whatsoever to do anything more for you as you've basically just agreed to become a burden for them forever.

Wow, a comment like this makes me glad I am with Linode. Way to not value all of your customers.

What's wrong with offering a solution? If you are having problems with that server, allow us to migrate you to another one. Sounds fair enough to me.

No, it sounds more like "you're costing us money now, so we don't give a crap about maintaining those servers anymore. If you really care about stability then pay us more money for our other service."

They should pull up the old contract, find out exactly what can be done to either terminate or transition people to the new services and get it over with. This seems like they're just letting the issue run off the rails and then when people complain that a server is down an extended period of time, a server that they paid a lifetime membership fee for, they start spouting off that these people are whiny freeloaders.

If the servers are old and rickety, migrate them to the lowest level of your new service for the same cost as what the person paid for initially, given that they've already paid in full, this would put the customer cost at $0. If you don't want to do that, then maintain the old servers and give them as much love as you would your other customers. That love includes not getting pissy in public about people whining over "free" servers, especially when they _weren't free_.

Jason, I remember you from back in the day... enjoyed your flickr series of photos where you blogged about setting up the then new Textdrive data center. That even got me to install open solaris and use it as a headless server for a while (and did not add to the downvote pounding you are taking here, FWIW). I don't see anything on the Joyent site about this upgrade offer to migrate off. In fact, I don't see a support section period. Do you have a link?

Hmm, Joyent migrated my TextDrive lifetime account to their newer stuff at no charge. Did you ask Tech Support if there was a migration path for you?

This is correct. We've been offering upgrades to newer stuff. You just need to contact support.

Is the upgrade free/reimburses for the old package?

Do we actually need to contact support to be upgraded? My understanding from the old Shared Accelerator documentation[1] (which you seem to have deleted) was that we were supposed to wait to be offered a “golden ticket”, unless we absolutely needed to be upgraded ASAP.

Are you even still offering regular (not “cloud”) web hosting? I can't find it on Joyent's site anymore; it seems as if the Shared Accelerator plans are discontinued, too.

[1]: http://wiki.joyent.com/shared:kb:start

Edit: Found it. This is the page that implies that TextDrive users have needed to wait for a “golden ticket” to be offered to them: http://replay.waybackmachine.org/20090305040858/http://wiki....

Ah-ha! Looking at the front page of their (apparently new) wiki, I found a link to the old wiki:

http://oldwiki.joyent.com/

And that wiki has up-to-date information on migrating from TextDrive to what are apparently now called “Shared SmartMachines” (formerly Shared Accelerators). And that page finally does resolve the question of whether we need to contact them or wait to be contacted: We do need to ask for a golden ticket, not just wait for it.

That's pretty well buried. Joyent really should send out a mass email to all TextDrive users asking us to upgrade, telling us exactly what effect it would have on our billing, and telling us exactly what we need to do.

It's still unclear whether, in the special case of Howe, we still need to ask for golden tickets, or all Howe users are being migrated to Shared SmartMachines without having to ask. The support post:

http://help.joyent.com/index.php?pg=forums.posts&id=949

implies that they are rebuilding Howe, which implies that we will not simply be migrated to a Shared SmartMachine as a result of the failure (we still would need to ask), but they have not directly said.

The one remaining worry I have is that there's nothing on the main Joyent site about Shared SmartMachines. It seems like they're inviting users of one deprecated service to migrate to another deprecated service.

Same experience for me. I have a lifetime account and they offered the migration to me for free. Granted, I had nothing but email and a vanilla CMS running so I had simple needs but the migration was very smooth.

Nope, they never suggested it and migrating never occurred to me (I assumed it would cost money).

I purchased a lifetime account when TextDrive put out its initial call for VC200 investors. That was many years ago, and I still feel like I'm treated well on the rare instances I need tech support, even though I'm still using one of TextDrive's original servers. They always encourage me to move to a Joyent Accelerator, and at some point I may, but I don't need much from my shared host and TextDrive/Joyent generally exceeds my expectations. Of course, a lengthy down time would change all that. I hope this serves as a wake up call for them and no one else has to go through what happened to the OP and others on Howe. My sympathies.

We never pressured anyone to repurchase anything. And switching from a shared hosting product from 2004 to a VPS at Linode. Makes sense but it's not a side-grade.

I bought a lifetime account as well, then migrated when offered: support and uptime has been the best ever for me.

The way to make the pain stop

No, the way to make the pain stop (and when I say pain, I mean hours and hours of downtime) is to work with a real managed hosting vendor that cares about your servers and your business, and to move your mission critical software off of commodity shared servers where it will get lost in a sea of customers, none of whom have an account executive, a sales representative, or assigned engineers who understands your stack from top to bottom.

Cloud services are great for some things like elastic computing power and rapid prototyping, but having my business' front door hosted on a service where I can't even call someone up in case of trouble seems like absolute craziness. I mean, how would you even implement a disaster recovery/business continuity solution when all the damn servers are in one building?

This has absolutely nothing to do with "Joyent Cloud". Nothing. The IaaS product has had 99.999-100% uptime this year. This is a 6 year old sharing hosting product that's currently free.

it's currently free? I pay $15 a month for it.

Only "free" for those of us who paid for a lifetime of service in advance when his company was starting. His gratitude is apparently nil.

Not sure that really matters. They offer a service, it was down, there was a shitty response. This is an indictment of the response to the downtime, not the fact that downtime occurred. Downtime occurs in every computer system.

Agreed downtime occurs in every system. Also, the most important function of a datacenter is to keep your application/service up and working properly. Use a third party service to be the first to know when it's down. (disclaimer: I am a founder of Rigor.com, and we provide this type of service.)

"Some products are better" isn't tenable. Regardless of whatever market segmentation you care to do, this extended outage is going to affect customers' perceptions of the diligence of your entire technical staff. If it's impractical to meet these obligations with the same quality as your other products, the thing to do is to buy the obligations out or sell off the unit delivering on them.

No it's not free, I pay $15/month, at least you have posted stuff unlike your support team who haven't responded to me for 48 hours now. Did I mention that this is the SIXTH day my domain and email that I PAY for is still down with no one telling me when it will come up. Don't tell me it's free just tell me when it's going to be fixed.

Holy crap! Fucking Howe! I was on that server three years ago for about two weeks before I could no longer take it. It was a misconfigured piece of junk when it was brand new. I've had a lot of success with Web Faction since then.

There seems to be some confusion in this thread about the difference between "free" and "paid for in advance".

Cutco will re-sharpen your knives for free, no matter how old they are, even if you bought them on eBay, because they made a 'lifetime resharpening' promise with the original purchase. And they're basically a multi-level marketing scam.

I have one single client who refuses to move off Joyent. My experience with them has been less than stellar. Frequent problems with no explanations, ask a support question like "How does one do X on your system?" and often get "oh you want to do Y I see, here is a link to the support forum for that." and the link isn't at all related to X or even the Y.

They aren't horrible, but I would never use or recommend them to anyone.

If you're on an old textdrive server, then you need to move, like we've been saying.

Have you considered offering to buy back the lifetime offers from people who won't migrate? It seems like that might be cheaper in the long run than having to support the old BSD servers forever.

As an upgrader to mixed grill, I agree. I moved and I've been happy. I assume at some point I'll be asked to move again from the shared smart machines as they currently exist (which, as near as I can tell, are identical to the shared accelerators), and I'm cool with that.

But because you (Jason) appear to be actively monitoring this thread, I'm going to make a suggestion:

E-mail everyone on the old BSD textdrive servers and tell them that they really should migrate.

Tell the month-to-month customers that you're end-of-lifing that product and on February 29th of 2012 (because nothing ever happens on Feb 29th) it will no longer be there. If you like, offer them whatever account is similar to the one they're on with a 50% discount 1 year discount if they take you up on it in the next month, 25% the three months following that and 10% up to the 29th.

For the VC customers just tell it how it is. You will continue to support the systems for however long you need to, but the newer stuff is just better. You appreciate their trust in your company etc. but they will probably be happier with the shinier stuff if they migrate.

Then follow up every couple of months with another e-mail. You could even make it interesting and say what percentage of VCs have migrated, what cool new stuff is available to those who have moved etc. (though if you do that, I'd like to receive those e-mails too).

If you want to incentivize the move for the stragglers, go ahead, their bitching pisses me off anyway and if you can get them to move and STFU by offering them something I didn't get, that's fine by me.

But most importantly send an e-mail. Don't post on the forum, or on the blog, or the wiki. I know that the forum, RSS feeds and blog are how Joyent likes to communicate but e-mail goes into my inbox. If I don't pay attention to it I'm the only one to blame.

If two or three years down the line people still haven't migrated. EOL the BSD servers. I'm pretty confident the "lifetime" I paid for was for lifetime of hosting with particular storage, server access, and bandwidth levels, not lifetime of hosting on a single machine with a particular OS. I could be wrong, but I think it's reasonable to assume that your hosting provider will change technologies as time progresses. I also am happy if they keep on giving me higher quality service with no reoccurring fees.

Completely incorrect statement, I never received an email to migrate.

http://www.joyentcloud.com/about/policies/cloud-service-leve...

For clause 2:

  2. Service Level

  Goal: Joyent’s goal is to achieve 100% Availability for all customers.

  Remedy: Subject to Sections 3 and 4 below, if the Availability of customer’s Grid Container is less than 100%, Joyent will credit the customer 5% of the monthly fee for each 30 minutes of downtime (up to 100% of customer’s monthly fee for the affected server).

So probably you can get this month for very little (Takes only ten hours for getting full refund if SLA is signed)

But at the same time their exceptions list are pretty long, things like emergency maintenance and upgrades are considered as exceptions.

Sounds like the OP is getting the service for free anyway...

It wasn't "free". We bought a "lifetime" package. I don't recall the details, but I ponied up a couple hundred dollars at a time when we didn't know textdrive was going to make it. I don't expect a lot of support, but I do expect uptime and courtesy.

Oh I see... then, I just wonder how they could eat up for life-time subscriptions for so long. After some Google searches, I've found that those plans are one off, and seems to cost $799 to sign up (http://hostingfu.com/article/joyent-lifetime-accelerator). If the terms was not changed, it now cost $25 a month on JoyentCloud to have the same thing with the SLA I mentioned. So it takes 31 years to compensate the cost of the $799 lifetime subscription to pay off. But the fact is that now they have "10 TB Monthly Bandwidth" (or unlimited depending on which web page you are looking at), I am just blindly guessing that lifetime users would get that too, right?

FOR JASONHOFFMAN

I see you posting quite a bit here...you keep saying the same thing: "take the free upgrade that is being offered".

There also seems to be quite a few old textdrive lifetime members here.

Perhaps the best approach would be for you to outline the steps for those textdrive folks to migrate off of those servers and take the free upgrade you are offering. I personally have no idea how...so if you could explain it, give a link or some concrete information it would be helpful.

I for one would be happy to migrate if it means better servers, I just need to know how...I'm sure I'm not alone.

From one of the very many forum posts on the exact topic:

"If you have not sent in a request and would like to get a jump on the migration, please send an email to support [at] joyent [dot] com with the following information:

- Subject line: migration request + plan type if known (startup, plus, premier) - Full Name - User Name(s) / Primary Domain(s) (please note new info if you want to change it at this time) - Existing Server(s)

If you wish your username or domain name to be changed, please include this in the email and we will process when we send your golden ticket. Waiting until after your account is set up will incur a $50 fee and you will receive a brand new account (old one will be blasted).

Please do not send follow up requests as it only slows the process down. If you asked questions when you sent in your migration request, we will answer those when we send your golden ticket out.

Once you receive your 'golden ticket', you will have 60 days to migrate to the OpenSolaris environment. If you need the second account for more than 60 days, regular monthly charges will apply."

Thank you for that.

I find it a tad bit odd that I received NO emails on migration. Frankly, there is no conceivable way that announcing in a Joyent forum and expecting users to become aware of it is a sound strategy. Pull vs Push and all that.

Even so, I'm willing to migrate if I can find some additional information as to what is involved.

Again, thanks for posting...I'll go digging through Joyent's forums now....

Haven't read all the comments but this really seems like a 'special case' being an exception and does definitely not apply to the experience and quality you'll usually get on Joyent.

I have been with them for a couple of years and the only reason why I left was because the project I used to host there was put on hiatus and I don't need that kind of 'enterprise' hosting for my other stuff.

I agree there shouldn't be downtimes on any of their machines but just going by the headline, you'd think Joyent is unreliable when indeed their own server plans are rock-solid.

You make it sound like they are Dreamhost. That said, hope your problems will get solved soon.

I will edit my original post as soon as they solve the problem. No response yet, other than excuses and denigration from Jason Hoffman (Joyent founder) in this thread.

Edit: I can no longer edit the original post, but I will post a comment.

We're Joyent customers as well, and their support has always been great, so this is definitely surprising. You should get on the phone with them (they provide an emergency number). What kinds of machines are you running? Is it just one, or many? It shouldn't be necessary, but message me if you're still not getting through and I can ping some contacts.

Thank you, we actually take our support very seriously and we're pretty accessible. He's talking about a different product than what you have.

No you don't, I asked via a support ticket two days ago for an estimate, I can't even get a response from the support team. S I am frustrated enough to post here

I have a lifetime account as well. Joyent's tech support has always been excellent. I take this as a wakeup call to get my stuff off of the old severs beyond that I can see little fault with Joyent.

btw. Elapsed time between opening the support request and an answer from someone at Joyent - 9 minutes... Elapsed time from opening support request to receiving account details for an account on a new machine about 26 minutes. That includes the time it took me to get back to them about some details of the account.

I'm sure I'll get down-modded for saying this, but on WHT people will tell you that you get what you pay for, which is in stark contrast with what most people here are saying (lack of gratitude for being a charter customer, etc).

Sure, you paid $200. But if you've been a customer since 2007 then you've basically paid the same price as any cheap shared hosting provider. Contrast this experience with Layered Technologies that ran three price hikes and then when I still didn't leave the server I had with them essentially told me to fuck off once their harddrive failed.

At least jasonhoffman had the courtesy to tell people their best option is to switch servers to avoid trouble.

But, like they say, you get what you pay for. If lifetime support and stability is what you'll come to expect, don't pay bottom dollar prices up front.

Disclaimer: I am not a Joyent customer in any way.

The better option would be for Jason Hoffman to say that he stands by his offer & will ensure that those who took advantage of the TextDrive deal will continue to see the same level of service or better that they received back when they bought it. Advertise your new services fine, but don't try to upsell people anytime there is downtime & don't go around telling everyone that these people are whiners getting their service for "free".

Okay, I get that migrating costs time and that's definitely not free, but saying that Jason upselling is false. It's been repeated by a number of commenters here and it's starting to bother me.

I'm a lifetime Joyent (TextDrive) member. When Joyent started offering newer shared hosting accounts, they allowed me and all other lifetime members the opportunity to migrate. No new fees, no hassle. Just a nice, shiny new space. To me, it's a testament of their support that they didn't try and keep all the lifetime members stuck on the old FreeBSD system.

I also find it unrealistic to expect lifetime customers to never have to migrate from one server to another. Do you really expect the server to run for 20 years without ever having to do anything yourself? Seriously, take some responsibility for your site and move to the latest supported service.

So there are no monthly fees after migrating from the old TextDrive servers?

It sounds like the offers to migrate have either not been sent to all or the messaging has been inconsistent as to costs/ease with which this can be performed.

I had some issues with a Textdrive server recently as well and migrated over to one of their smart machines (at a lower price than what I had been paying). I was a bit frustrated like you, but Joyent was extremely, extremely helpful in assisting me with the migration and now everything is running smooth.

I would try to touch base with them again and see if they can help you out.

And there's no reason that you needed to pay additional unless you bought a different product.

I remember trying to get a quote / get a few details about buying large quantities of bandwidth from Joyent a couple years back.

They consistently took days to answer simple questions via email. I generally take it that if a company can't even get its sales people responding quickly there's a snowballs chance in hell it's techs will.

If it was a couple of years ago and for large quantities of bandwidth, then you would have been talking to me. We didn't have "sales people" a couple of years back. And I would have said "No" because we weren't in the business of selling "large quantities of bandwidth"

You get what you pay for.

This isn't a glib statement. A business transaction is about more than coming to agreement about the price of a thing or service. It's also about creating and maintaining incentives for both sides.

I pay slicehost for a VPS. They get $50/month from me in return for disk space, CPU cycles, uptime and support. Our incentives are aligned.

With Joyent/Textdrive, every month you get hosting and every month they get nothing. The incentives are not aligned. In fact, from Joyent's perspective, the incentive to provide any meaningful service diminishes over time. No about of cajoling or complaining or threatening Joyent will change this imbalance. Sure, it's 54 hours of downtime today - but I can guarantee that a year from now the outages will be far longer and far more frequent.

"But we had a deal!" Yes, you did. And you were a fool to enter into it.

Here's the bottom line:

Your customers don't care what's causing the downtime. It's still your fault.

Website users don't care that the deal has soured. It's still your responsibility.

Joyent doesn't care that they're providing cr@ppy service to Textdrive users, despite the postings on HN by Jason Hoffman (hey Jason, instead of wasting time posting on some internet forum, why not spend that time fixing the actual problem?)

Seriously - Joyent doesn't give a sh?t about the lifetime accounts. If they did, then why are you experiencing 54 hours of downtime? If they do care, and are applying resources to solve the problem, then why is it taking so long? 54 HOURS TO FIX A SERVER. That, to me, is evidence of a shocking level of incompetence. In either case I would get the hell out of there.

T minus 10 minutes before this hits TechCrunch

I've had a very bad experience with Joyent. Abysmal performance, tech support in denial in spite of obvious tests. Just leave, your life will be better.

I remember that chunkhost has really aggressive pricing.. And I remember there service being pretty good...

He's currently at "free", so get any more aggressive than that.

Paid for lifetime service in advance to help bootstrap your company, not "free".

Is there any time that Mr. Hoffman's going to call it quits on this thread and actually take it offline in an attempt to resolve the OP's issues?!

Frankly, the tone of most of his comments have been borderline denigrating and it's doing nothing but adding nuisance value to the thread.

He's probably getting exasperated by the virtual lynch mob which seems to be downvoting everything he says and upvoting every angry comment, which is ridiculous. He's actually saying quite a lot of reasonable and useful stuff which is sitting at 0 or below.

joyent had this sort of failure regularly about 4 years ago and continues to do so. Avoid.

Also incorrect. "About 4 years ago" we always did >99.99% uptimes even on the old textdrive stuff. I'd be happy to show all the old stats and graphs, we still have them.

Jason, you're experiencing a PR hiccup; it's not a big deal but it's also not clear that responding to every message in this thread is going to help you. Instead, why not (1) make things right for the OP and everyone else on Howe and (2) put a response on the blog?

You Joyent folks will never get a subset of your customers to migrate off textdrive hardware (so it's no use to keep yelling "just migrate!"). However, if you let those older servers fall apart you'll only get more negative PR.

I get having a passion for your product -- I really do. Some of the responses here from Jason are on the border line, though, and I think they are making the situation worse overall. I wish, for his sake, that he'd have quit while ahead.

Without a doubt, Jason has the best intentions here, but I'm getting a really bad vibe from reading the entire thread -- particularly the by the way, you contributed to a ticket and I can look it up so I'm going to remind you so that you'll look stupid for bitching about us bit.

The hostile you-are-wrong attitude, while maybe technically correct, is a disaster for outside perception.

If it were me? I likely would have EOL'ed those FreeBSD boxes a long time ago. Customers, even the life timers, should not have been given a choice. Why?

Textdrive promised lifetime hosting. Where does it say they promised hosting on the same box for that entire lifetime? TxD/Joyent has been gracious to provide hosting on their new platform.

One could argue that the 'lifetime' was referring to the actual box, and if the downtime is increasing, the lifetime of the server is ending.

I say pull the plug and force the migration. Granted, it could be a small PR hit.

I think he is referencing BingoDisk (perhaps I am mis-remembering the name of the service which was offered by Joyent).

Nice you have a backup and recovery in place for your stats, shame you couldn't have one in place for your server

I will be moving from joyent. I am paying 15usd per month and have not been told to migrate. now I read the founder saying it's my fault that I am suffering so much downtime because I should have searched the forums and migrated years ago. Well I am going to migrate now.

You must be on HOWE too. I am in the same boat - unhappy. send a tweet to CEO David Young @davidpaulyoung. (from @snapplez)

joyent is enterprise grade and doesnt have this sort of downtime. its probably a break in your app.

http://help.joyent.com/index.php?pg=forums.posts&id=949&... acknowledges it's a real outage and talks of rebuilding the server after a significant hardware failure from two days ago, which implies they didn't have a disaster recovery plan.

Good catch; I just wonder how it can take more than 2 days, when you already have a bunch of employees as well as equipment and presumably spares, to build a replacement FreeBSD server and transfer data?

The lesson jasonhoffman could learn:

If you have a small set of users who bought lifetime accounts to bet on a company you founded than even with new company I would bend over backwards to make sure their needs are met as a thank you.

Otherwise some gets the bright idea to put up a joynet sucks web page with proper Google approved SEO and makes this little PR bump a big EFFING MOUNTAIN!'

jasonhoffman, I imagine you have less than 12 hours before that happens..as TechCrunch may do it for 'free'..:)

Parse Error.

Digging Techcrunch got me this story: Twitter and Joyent splits (Circa Jan 31, 2008)

http://techcrunch.com/2008/01/31/twitter-and-joyent-split-am...

Twitter moved to Verio and here's Biz Stone's reply to Techcrunch:

We’re still very much engaged in our efforts to bring solid reliability to Twitter. Achieving our goals is a sustained effort, not an overnight fix. Performance is our most important measure of success and we appreciate both the patience and frustration folks are sharing with us.

With regard to discussing technical specifics about last night’s efforts, we’ll be more keen to do that once we have a chance to come up for air and cover it with some perspective.

Ouch.

It sucks that your hosting is down, but this is a pretty malicious use of HN considering you paid $200 over 6 years ago for lifetime hosting. Less than $3 per month by now. What SLA were you guaranteed?

The lesson for service hosts, NEVER offer lifetime anything.

Reply on news.ycombinator.com