RyanGWU82

Amazon rebooted lots of PV guests. Presumably they collocate HVM and PV guests on the same box. If there were any HVM guests on the box, then there could be the possibility of an attack. (I guess they could forcibly kick off the HVM guests, but that wouldn't be very nice.)

Why shouldn't Netflix be running on HVM?

I was pretty unhappy with how Rackspace handled the maintenance window.

1. Rackspace's maintenance announcement was sent at 9:00 PM on Friday night (Pacific time). Seriously?! I had already left for a weekend vacation without my laptop, so I couldn't do anything to get my company prepared. Even if the patch wasn't ready until Friday night, Rackspace could have scheduled the maintenance windows and announced them to customers much earlier.

2. The maintenance window for all three USA regions were scheduled at the same time. We couldn't just move to a different region without going to another continent.

3. Each maintenance window was 24 hours -- that's just too long. Even though our servers were only down for 10 minutes, we had to be on call and ready for 24 hours.

4. Although we have redundant servers in every region, we still couldn't guarantee that those redundant servers wouldn't be rebooted at the same time. As it turns out, we did lose both of our servers in ORD at the same time.

I'll look into it and let you know!

We push DNS into Route 53 at Runscope, and I trust our approach for the reasons you mentioned. Route 53 has a great track record for DNS uptime. If our sync system breaks, Route 53 will keep serving DNS records while we fix the sync system.

On a side note, we actually have two DNS sync systems at Runscope. One handles DNS for individual hosts (like web01.runscope.com) and the other handles public DNS (like www.runscope.com).

Our public sync daemon is tied into ZooKeeper, which keeps a list of active hosts for each of our public services. The daemon updates entries in Route 53 whenever the hosts change in ZooKeeper. We have 3 different types of public domains, and the tool handles all 3. We have standard domains with load balancing, geo-routed domains for services in multiple regions around the world, and regional domains for routing to a specific region. The sync daemon has been very helpful in keeping our infrastructure nimble.

If this interests you, I'll be giving a talk about our cloud infrastructure in a few weeks in San Francisco, along with our founder/CEO John Sheehan. Full announcement coming soon...

Detail Page (page of product details).

I used to work for a pre-IPO startup that had experienced a pretty big run-up in their stock price. As a result, their new hires were getting options to buy 200 shares at $100 per share. (I made up those numbers, but you get the idea.) They did a big stock split partly to change the numbers in stock option offers. Apparently it was much more enticing to offer candidates options for 2000 shares at $10 per share. Even though the options would have the same total value, they believed candidates would respond much better to an offer of "more options."

This post was just an incident review for a technology audience. Dropbox posted a separate apology to their users on their main blog: https://blog.dropbox.com/2014/01/back-up-and-running/ . The tone and detail seem totally appropriate since it ran concurrently with the other post.

I'm in the same position, although I returned my kit before the November 22 cutoff date. I figure I could probably request a refund given the new FDA situation -- given their regulatory situation, they'll probably loosen their refund policy to avoid compounding matters. But I don't think my kit should have any less information than before, so I figured I'd let it play out and see what the results look like. Obviously if my results don't include health data, then I'll certainly ask for a refund, because that was the main reason I enrolled.

Is there any service provider whose TOS compels them to provide service unconditionally to the general public? If you were running an SaaS service, wouldn't you want some recourse if a customer was abusing your service?

If you're a large customer and you're really concerned, you can probably negotiate a contract with GitHub that guarantees your access to the service. But if you're a large enough customer to warrant a custom contract, then no rational service provider would unexpectedly cut you off, because your business is valuable to them.

I'd compare Bose to a full-size sedan, something like a Toyota Avalon. Toyota designed the Avalon to provide a smooth and calm ride in a luxurious package. So the Avalon's suspension/steering/engine are all tuned for a smooth ride, which is the exact opposite of what car enthusiasts want. An active driver would be happier with a BMW or a Subaru. But if my grandparents drove a Subaru WRX, they'd hate the same qualities that make it enjoyable to the enthusiast.

Bose actively processes the input, which does improve the sound quality to the casual listener. It's exactly what those kind of listeners want. Audiophiles want exact reproduction of the original source, so it's not a good choice for their preferences. But by dissing Bose, audiophiles are doing a disservice to the casual listener, the same way as if I pushed my grandparents toward a WRX.

S3's not completely indestructible, but it is the most reliable platform I've encountered. During the ~1.5 years I was at Pinterest, we used S3 to store and serve all of the cached images on our site, with CDNs in front of course. We only ran into 2 incidents with S3, and only one of those would have affected "normal" customers. (The other was our own fault -- we suddenly started writing hundreds of files per second to a bucket that wasn't ready for this level of traffic.)

Compliance is the main reason to purchase Dedicated Instances -- maybe the only reason at all. Amazon could always silently introduce better physical hardware, at which point they could put 2+ cc2.8xlarge instances on a physical server. I'm sure there are more m1.smalls on each 2013-era box than there were in 2007.

Both of these features look really useful; kudos to AWS for launching them. I've already moved my personal website's root domain directly to CloudFront. (I was previously hosting the root domain through S3 and the "www" through CloudFront, so it's nice to have them both set up the same way now.)

Have you tried S/MIME instead of PGP? S/MIME support is built into desktop mail clients like Thunderbird, Microsoft Outlook, and Apple's Mail.app.

I think the biggest direct competitors are Rackspace, Windows Azure, Google Cloud, HP, and Dell. Indirectly, VMware and OpenStack are huge competitors too. Lots of companies are trying to decide how their existing infrastructure fits in with cloud computing, and the "private cloud" is one likely choice.

None of these have the wide product functionality that AWS has -- yet. Amazon has a multi-year head start on most of these companies, but as you mention, there's lots of time to make progress before 2020.

My company used gevent in production at very large scale, and we were extremely happy with it. In fact, we ported our existing Django and Flask applications to run under gevent, which was a surprisingly fast process. (Weeks, not months, to port rather large codebases.) We did have to be careful with third-party libraries, like Zookeeper clients, but that was worth the tradeoff. We got the performance of an evented structure without having to rewrite a ton of code.

Exactly -- I was benchmarking a disk that was having a performance problem. and I did something like "dd if=/dev/zero of=/dev/sda" instead of "of=/testfile".

/dev/zero is plenty fast... :-/

I have my own domain for email, IM, and my personal website. It allows me to move from one provider to another without having to update my address everywhere. Even if Google doesn't shut down Gmail, something better could come along, and I don't want my email to be tied to Gmail forever.

It's not so funny when you do this accidentally, on a 2 TB disk full of user data, on a live production server, at 2AM on a Friday night...

(The server was part of a MogileFS cluster so there were multiple copies of the data online. There was no data loss, not even any downtime. Still, it was scary as hell, and I spent all Saturday in the data center restoring the box.)

This isn't getting much attention on HN, but this is a Big Deal.

It's important to be on Virtual Private Cloud for a few reasons. Here are a few:

1. VPC gets you out of the EC2 private IP space, and lets you configure instances without public IPs either. This helps prevent accidental misconfigurations, like accidentally letting the whole world connect to your Memcache.

2. If you want to connect a VPN to your physical infrastructure, or to connect servers across multiple regions, VPC makes that much easier.

3. Amazon's stopped launching new networking features on standard EC2, so if you want to do things like change the security group of a running instance, you can only do that in VPC.

The problem is, most startups don't think about VPC prior to launching, because these things don't matter much when you're just starting out. But unlike most early ops shortcuts, it's a huge pain to migrate to VPC later. It's basically the same effort as moving data centers, except for actually carrying servers around in your hatchback.

Now new customers won't have to think about VPC upfront, but also won't have to go through a messy and time-consuming migration. Although this won't help existing EC2 deployments, this is a huge win for new AWS customers.

Why do we conduct multiple interviews in the first place? Because, as a group, we want to gather all the information we need to make an informed decision. Some quick discussion between interviews can help make sure we gather all the information we need -- as long as you keep it to narrowly defined topics. In particular, I always try to find out: "What did you cover in your interview? What didn't you cover? Is there anything I need to know?"

Let's say we're interviewing someone for a role with HTML5, CSS, and JavaScript. If the first interviewer ran out of time before getting to JavaScript, I should probably start with that.

Or, if an interviewer noticed any red flags, we can use the following interviews to gather more information. If I was told that a candidate complained loudly about being overworked, then I could spend more time discussing time management and the way he works. Maybe he legitimately was overworked, or maybe he's a complainer, or not very good at prioritizing his work.

Finally, it helps if there's at least one recruiter or a manager that you can give real-time feedback. Recently I was on an interview loop of a candidate for a very specific role. I quickly realized she was a terrible fit for this position, but she'd actually be perfect for a different one. I went to our recruiter and explained the situation. He was able to reorganize the schedule so that the candidate spent the rest of the day interviewing for the right position. If we had just continued without any feedback, it would have wasted a lot of time and we would have missed out on a great candidate for the other role.

Prior art is irrelevant for trademarks -- a trademark doesn't have to be unique for you to use it. There was an item called a "macintosh" before the name Macintosh was trademarked. The word "dropbox" had been used in file sharing software long before the YC startup existed. Even the creator of gaymer.org doesn't claim to have invented the word "gaymer."

You can claim a trademark as long as no one else has already claimed the same name in the same context. Once you have established the trademark, you can prevent others from using that name, even if they have been using it for a long time.

Note that the "same context" is why there's unlikely to be a conflict with Gaymers Cider. In particular, the gaymer.org trademark only applies to social networks.

(IANAL, so I may have some details wrong here.)

We revise the "right number of instances" every few weeks based on latency and traffic numbers. But sometimes when we release updates, we'll find that we suddenly need a lot more capacity (or a lot less if we improved performance). We have automated tools to help us notice performance regressions. Once we decide that we need to change the pool size, we adjust the watchdog configuration by hand.

Our #1 requirement has been to keep up with the growth in traffic on the site. We've been growing so fast that there's literally no way we could have ordered and racked equipment fast enough. We were also a very small team -- a year ago there were only about a dozen people in the whole company. At this point we're much larger, which gives us room to consider more options like colo or multiple cloud providers.

AWS certainly feels pretty costly when you compare colo prices to the list price for on-demand instances. But one of the reasons I wanted to present our work is to show that you can use the cloud for a lot less than the list price. It takes work to buy reserved instances or run spot instances, but that does make it much more cost competitive.

Our watchdog process calls the EC2 APIs directly to identify how many instances are running, which ones are spot instances, etc. Boto, the AWS client library for Python, makes that pretty easy. The watchdog isn't very sophisticated -- it just checks to make sure that the correct number of instances are running in each auto-scale group. Our application servers aren't very efficient in certain respects, so we don't trust metrics like usage/load to make auto-scaling decisions.

If I was doing it over again, I'd just use Amazon's auto-scaling features for all of this. At the time we built this, EC2's auto-scaling didn't support some of the features we needed. Since then, they've made it a lot easier to do things like set up a repeating schedule for auto-scaling, rather than using metrics.

We only have one EC2 AMI that we use for all of our servers. That AMI is pretty basic; it only does enough to connect to our Puppet configuration management servers. Puppet then configures the boxes as web servers (or databases, or...) and adds them to the appropriate load balancer.

Right now we run everything in the US-East region, and we have all our services balanced across 4 availability zones. If there's a problem in a single AZ, it will affect every layer of our system, but only about 25% of the hosts in that layer. Some of our services are automatically resilient and can handle that easily. Others aren't so great, but we're working on more automatic failover.

When we need more servers for an auto-scaled service, we open spot requests and start on-demand instances at the same time. For most services, we want to run about 50% on-demand and 50% spot. We have a watchdog process that continually checks what's running. It launches more instances whenever there aren't enough, and terminates instances when there are too many. So if the spot price spikes and a bunch of our spot instances are shut down, the watchdog will launch replacement instances on-demand. It will also request more spot instances once the price has dropped back to normal. In reality we don't often run into spot capacity issues -- maybe once a month, and it's almost never apparent to our users.

I spoke about this in detail at AWS re:Invent last month, and the full talk is available online here: http://www.youtube.com/watch?v=73-G2zQ9sHU

I'm Ryan Park -- I'm the Pinterest engineer quoted in the article. I'm happy to answer any questions about our setup.

Just to clarify, the auto-scaling was specifically for a pool of web application servers. At the time I gathered the numbers, there were 80 servers in that pool. In the last few months we've been moving toward a service-oriented architecture, and we've been able to use the same code to auto-scale the internal services. Of course it's not possible to auto-scale stateful servers like databases, but it's still saving us a considerable amount of money.

We implemented the auto-scaling in early 2012, so it's been in use for almost a year now. It only took about 2 weeks of engineering to build the system. It does need occasional maintenance, but it's still worth the effort given how much money it saves us.

I think you may be mistaken -- under "Projecting Costs" that page says that it's 10 cents per provisioned IOPS month.

I run dozens of memcache and hundreds of app servers in EC2, and we haven't seen any evidence of connection throttling. Isn't this just network latency?

If your PHP code can make 300 requests per second, then each request takes about 3.33ms. That's actually quite good if it's going over the public Internet to another data center. If you're getting ~10 requests/sec. across the country, that's about 100ms per request, which again is typical of cross-country latency.

If you need response times faster than that, you need a direct network connection between your app server and your memcache server, and you definitely need to avoid transcontinental network hops.