mmalone

If you're a net neutrality advocate you should seriously consider whether this sort of peering arrangement should be tolerated. This kind of peering is a de-facto QoS arrangement that gives Netflix a performance and cost advantage over other video providers that aren't big enough or rich enough to justify and pay for direct peering with ISPs. It also gives bigger ISPs (that do more than 2Gb/s of traffic to Netflix and agree to peer, apparently) the same advantage over their smaller competitors. Isn't that exactly what we're trying to avoid with net neutrality?

This is not a new practice and I'm surprised it doesn't come up more frequently in the net neutrality debate. It's been going on for years -- broadcast.com did it, YouTube and Google do it, and I'm sure others do as well. We've never really had net neutrality -- peering agreements have been happening forever in shady back room deals at NANOG. At least the more recent shenanigans are happening out in the open.

This is the alternative means of entry page for a contest. This isn't even a page that Apple _wanted_ to build. It's a legal requirement. The CAPTCHA is provided by a third party. Who knows why it was chosen. I doubt they put a ton of thought into this page, or the choice of CAPTCHA, if they even built it themselves at all.

San Francisco, Full Time, H1B is OK

Betable is hiring awesome software engineers.

About Us

Betable engineering is an elite team of hardworking dreamers and tinkerers. We’re pragmatic, but we don’t let that interfere with our perfectionism. We’re fluent in computing concepts from anonymous memory management to monadic zeros.

We’re building a platform. Engineering innovation is our competitive advantage. We’re doing something new and different in an industry that hasn’t seen any real innovation in the last century. That excites us.

We try to be technology agnostic, preferring suitability to ideology. Our services are small and chatty, communicating through well-defined interfaces. Sometimes we compose them in ways that are surprising even to us. We think that’s healthy for a platform company. In general, we believe that sound system architecture and design are more important than any particular technology.

Day to day we use Node.js, Cassandra, Jenkins, Go, Java, Puppet, and GitHub. We like continuous integration. In fact, we like most forms of process automation. Sometimes our tools make us happy, sometimes they make us sad. When they make us sad, we make them better.

We like to work. We love to play. We don’t consider these activities mutually exclusive.

There aren’t many of us. We’re looking for more.

About You

You’re an engineering powerhouse who wants to work on challenging problems with a group of equally amazing colleagues. Your depth of knowledge in many areas of computing should qualify you as a specialist, but your breadth of expertise demands that you describe yourself as a generalist. Sometimes you think the two aren’t so different, but you resist that sort of reductionism.

You’re well rounded. You have hobbies that don’t involve computers. Despite your expertise, you sometimes surprise casual acquaintances with your profession. You can’t carry a tune but you sing anyways. Your recreational reading includes works by Kurt Vonnegut, W. Richard Stevens, and Dr. Seuss.

You’re not sure whether P equals NP, but the possibility leaves you feeling both titillated and terrified. Sometimes, while considering the repercussions, you lose composure.

Mutable state is your frenemy. You appreciate what Scala has managed to do on the JVM. You’ve developed several profiles of the language, each Turing complete. You’re curious about the future of Go. You’ve toyed with several syntactic variants of your own competing language based on the π-calculus. You’re an expert at SQL, but would never describe yourself that way.

You had a check for $2.56 from Donald Knuth. You cashed it when your bank started displaying digitized check scans in your transaction history because you hate physical media.

You can explain technical concepts to non-technical people. Non-technical people enjoy the experience. At parties your conversations transition seamlessly from Emacs vs. vi to Beatles vs. Stones.

You consider coding a craft – as much art as science. It’s your creative outlet. You consider quines a form of poetry. You often describe code as beautiful.

You’re self aware. Some of these statements don’t apply to you, but you can read between the lines. Others upset you. You understand why at a fundamental level. You can argue your position, and you can win.

Sound like you? Join us.

http://betable.theresumator.com/apply/KYLS0t/Software-Engine...

Not sure where you're getting your information, but it's almost entirely wrong. GISelle is the same tech as Penelope. It was renamed because there was a trademark issue. Polygon search has been implemented for some time, but was delayed for product reasons (it's actually live, just undocumented). Tag indexing was launched with SimpleGeo storage from day one. Indexing of all properties by default happened several months before the acquisition. Both features were developed by people who are still at the company.

The company was acquired because it has real, defensible technology. If you're going to talk shit, at least get your facts straight.

@foobarbazetc: you seem to know an awful lot about the SimpleGeo acquisition. Which VCs were forcing Urban Airship to do the deal?

The argument in the article that inspired this work[1] (and it may be an academic one) is that data models are all semantically very similar. The authors argue that category theory provides mathematical formalism for a generic abstraction that would bridge both the relational and K/V worlds.

The ideas all seem very formative to me. There are a bunch of assumptions, logic gaps, etc. But it's not all bad.

[1] http://cacm.acm.org/magazines/2011/4/106584-a-co-relational-...

If this is the first time you've heard about a kernel problem on EC2 you're probably not managing a very large EC2 infrastructure [1, 2]. Even in non-virtualized environments, at scale, it's common to run into linux kernel bugs, or at least peculiarities. Which is why large tech organizations invariably employ kernel dev teams.

The guy who claimed EC2 kernels are linked to the distro you run was simply claiming that, unless you want to go it on your own, you're tied to the kernel provided by a supported AMI. As you've suggested multiple times, there are benefits to running an environment that is supported and that other people have operational experience with. Honestly, I'm not even sure what you're arguing anymore... seems like you're just being antagonistic.

[1] https://bugs.launchpad.net/ubuntu/+source/linux-ec2/ [2] https://bugs.launchpad.net/ubuntu/+source/linux/+bugs?field....

Amazon's official linux build, at this time, is a custom distribution that uses RPM and is different from the Ubuntu/Debian world in several significant ways (e.g., a new libc implementation). Migrating your WordPress blog to a new platform might be easy, but when you're managing hundreds of machines running thousands of packages that sort of change is not trivial.

Running "time-tested" kernels is not really the best advise either in this case. Xen is a fairly new environment, and EC2's implementation has some quirks, so there's a pretty regular stream of bug fixes and other improvements in recent kernels that are often worth picking up. If I went to Canonical with a "time-tested" kernel bug they'd tell me to upgrade before they'd give any real support.

When we talked to Amazon about switching to their AMIs they advised us that it was probably _not_ worth switching, that switching might not fix the problem, and that the AMIs we were running were widely used and supported. They made it clear that they work closely with Canonical and other providers to get high quality AMIs into their ecosystem. Long story short, the people who you admit know the most about the EC2 environment advised us that they weren't necessarily the best option, or at least not the only option, for good AMIs (sort of like how hardware manufacturers aren't the best option for an operating system).

So the answers aren't really cut-and-dry here. Every time Amazon changes their dom0 there's a chance your "time-tested" kernel will stop working. And just because Amazon runs the infrastructure doesn't mean they're the best choice for a Linux distribution.

This bug affected several "supported" AMIs running 2.6.32 series kernels that we tested at SimpleGeo, including the official AMI released by Canonical. After we ran out of patience debugging this stuff we contacted Amazon and worked on the issue with a guy from their kernel team (who was really helpful, fwiw). He agreed that the behavior was bizarre and opened an upstream bug with Canonical [1].

You're sort of contradicting yourself here. You suggest that the distro you're running is independent of the kernel version you're running. But then you go on to claim that this bug was introduced by someone who was not running the default supported kernel. Are you saying that people should run the supported kernel, and be tied to whatever's supported upstream, or are you saying they should risk building their own? Clearly there are benefits and drawbacks either way.

[1] https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/708...

Or even FreeBSD, which works on EC2 now. It uses a different technique for scheduling, so this issue will not affect it.

BigTable enforces integrity constraints defined by a schema on a per-row basis? This is interesting news. Why hasn't Google included this information in any of the published materials describing the system?

- Forever Malone

Credit where credit's due, the 10gen and MongoDB guys have done a great job convincing developers to adopt their product despite the existence of technically superior alternatives. I guess that's what happens when a bunch of ex-DoubleClick execs start a database company.

I run a database that does not fit in memory on AWS with great success. Generalizing is dangerous. If you work with and understand the constraints imposed by the environment you can do some pretty amazing stuff.

Disclosure: I work at SimpleGeo.

You're right, copyright does not protect facts _in the United States_. Rules differ around the world. In the EU, for instance, there's a "sweat of the brow" criteria for database protection.

Even in the US, though, copyright law is largely irrelevant. When you sign up to use a web service you generally accept terms and conditions that lay out exactly how you're allowed to use the service and the data you consume. From that point on your interactions with the service are regulated by those terms under contract, tort, and other common law statutes. If the terms say you're not allowed to redistribute the data then doing so would be violation of contract and you would be liable for damages. Basically, things get really messy really fast.

I'm an engineer, not a lawyer. So, honestly, I'm not an expert on this stuff. But I think most engineers agree that the current situation is sort of stupid. Philosophically I agree with the copyright law as it stands. Facts should be free. So CC0ing the data is just an explicit, formal, contractual statement that we're not limiting how our data can be used anywhere in the world.

My point was that for an interactive application it doesn't take a very long time for user experience to degrade substantially.

And we've glossed over my points about newer databases exposing these knobs to clients. It's possible to do exactly what you're describing using Cassandra, for example (er, you might have to do some consistency level tweaking to make writes fail if you can't get a quorum of authoritative nodes, but it wouldn't be hard - and I'm not even sure if that's necessary). It's not possible to do it with MySQL or PostgreSQL without building some intelligent partitioning layer on top. And that layer will probably make it impossible to do joins and add relationship constraints, so you lose any benefits these systems bring to the table.

Uhm, of course they're not the same thing... but they have the same effect. The point is that the system remains available even if a node becomes unavailable for _whatever_ reason. I'm not sure what you're disagreeing on... There are common and uncommon modes of failure. Of course we should prioritize handling the common ones. But if we can handle all of them at once that's ideal. And, as I said in an earlier comment, when you're doing a million operations a second, failures that are one-in-a-million happen every second.

Your first point is not true. In the absence of partitions, all "eventually consistent" data stores that I know of will give you strong consistency. The eventual consistency bit only comes into play if a problem occurs. (It's probably also worth noting that many RDBMS replication strategies won't give you strong consistency at all - even in ideal circumstances.)

I suppose you could retry if a write fails (e.g., can't reach a quorum), but you could theoretically end up retrying forever (and 10 seconds or so is forever as far as an interactive user is concerned)... eventually you need to either fail or write inconsistently. So we're just delaying the inevitable.

Also if you're accepting quorum writes to the "major partition" you still have to repair the "minor partition" when it comes back online. There's no traditional DB that implements the sort of read-repair/hinted-handoff/anti-entropy mechanics that Cassandra, Voldemort, and the various proprietary big data stores use.

Er, no. I'm just not differentiating between the various reasons why a single node may be unavailable. It doesn't really matter _why_ the node is unavailable... it just is.

FWIW, databases like Cassandra expose the consistency tradeoff to the client. You can do quorum reads/writes with Cassandra. You can't with MySQL or PostgreSQL.

Edit: you can choose between quorum reads/writes and stronger or weaker consistency levels with Cassandra, but can't with MySQL / PostgreSQL.

It _may_ be a worthwhile tradeoff. But if you're in three datacenters and one of them splits you could have a third of your requests fail. Depending on what you're doing, that may be unacceptable. I agree that it's a tradeoff. I disagree with the people who are saying that emphasizing consistency is always the right thing to do.

Sure, but quorum protocols only provide strong consistency in the absence of partitions. If a partition occurs you may not be able to get a quorum (where R + W > N) and, again, you're stuck with either being unavailable or potentially inconsistent. There's really no way around it... AFAIK it's a logical impossibility.

I'm not sure I get your argument re: CAP being overapplied. The key point the whole "AP" camp is making is exactly what you're saying - "just because some fraction of your nodes are offline, that doesn't mean the whole system is offline." What it does mean, though, is that some of your data may be stale. But eventually it won't be.

As for WAN partitions, I agree, they're not as frequent as single node failures. But as far as CAP is concerned it doesn't really matter. A partition is a partition, whether it's one node or half your cluster. The frequency that "WAN partitions" occurs depends on how you define a "WAN partition." If you consider a single lost TCP connection a short-lived partition (it pretty much is), or if you consider a DNS or power outage a WAN partition (in the sense that a whole cluster might disappear) then I think we can all come up with lots of ways WAN partitions can and do occur. I do agree that the entire Internet doesn't go down very often.

Actually he's not really addressing it at all. What stats does he have that show WAN partitions are rare? I find that short WAN partitions occur regularly. And how does the small portion "block?" How does it even know that it's the "small portion?" If a read comes into the "small portion" (maybe because the client's in the same data center) what does the small portion do? If it responds, the response may be inconsistent. If it doesn't, the data store is unavailable (at least as far as that client is concerned).

I posted this comment to the ACM article, but they're pre-moderating and apparently someone's shirking their responsibilities... so I'm reposting here. Hope you guys don't mind:

Hey Michael,

Interesting read. I'm with you on most points. I do have a couple comments though...

Failure mode 1 is an application error.

Failure mode 2 should result in a failed write. It shouldn't be too hard to trap errors programmatically and handle them intelligently / not propagate them. Of course the devil's in the details and hardware / interpreter / other problems in components that are outside of the DBs control can make things more difficult. These are the sorts of issues that tend to stick around until a system is "battle tested" and run in a couple of large / high volume operations.

Failure modes 3, 4, 5, 6 (partition in a local cluster) - this seems to be where the meat of your argument is... but you totally gloss over your solution. I'm not sure I believe that network partitions (even single node failures) are easily survived by lots of algorithms... Or, more specifically, I don't believe that they can be survived while maintaining consistency (in the CAP sense, not the ACID sense). I threw together a super simplified "proof" of why consistency is essentially impossible in this situation in a recent talk. See http://www.slideshare.net/mmalone/scaling-gis-data-in-nonrel... - slides 16 through 20. What algorithms are there to get around this? If a replica is partitioned you either can't replicate to it and have to fail (unavailable) or you can't replicate to it and succeed anyways (replica is inconsistent).

I also don't buy the argument that partitions (LAN or WAN) are rare and therefore we shouldn't worry about them. For a small operation this may be true, but when you're doing a million operations a second then a one-in-a-million failure scenario will happen every second.

Failure mode 7 will probably result in some data loss unless (as you mention) you're willing to live with the latency of waiting for durable multi-datacenter writes to occur. But having that option is definitely nice, and that's a trade off that I'd like to be able to make on a per-write basis. I may choose to accept that latency when I'm recording a large financial transaction, for example. Another thought related to this issue - in a lot of ways writing something to memory on multiple nodes is more "durable" than writing it to disk on one. So you may be able to do multi-DC replicated writes in memory with tolerable latency assuming your DCs are close enough that the speed of light isn't limiting. That should get you durability up to the point where the entire eastern seaboard disappears, at least.

Failure mode 8 is another core issue that I think you're glossing over. WAN failures (particularly short-lived ones) can and do happen on a regular basis. It's true that routing issues are typically resolved quickly, but it's another law-of-large-numbers thing. Amazon AWS had an issue that took an entire data center offline a while back, for example. Shit happens. In CAP terms this is really the same thing as a failure modes 3, 4, 5, 6, and 7 though. So the same arguments apply. Re: your argument that only a small segment splits - what happens when a read comes into the small split segment (maybe from a client in the same datacenter)? If data has been updated on the larger segment it couldn't have been replicated, so again you're either serving stale data or your data store is unavailable.

Thanks for putting this together, it was an interesting read. Looking forward to hearing more about some of these issues!

Up and coming Django developer lets fame go to his head. It's poeple like this that ruin communities! Wtf??? So glad I chose RAILS!

Slides are also on slideshare at http://www.slideshare.net/mmalone/scaling-django-1393282 if you're into that sort of thing.

Free booze? I'm in.

I think the idea is to add some functionality that would make the browser a viable environment for "sovereign applications." That is, applications that are long running and highly interactive. Ajax has gone a long ways towards doing this for the web, but a local persistent data store, real networking, offline capabilities, and more fine-grained control of the UI (drag & drop interface, editable, etc.) will make a huge difference.

I wouldn't say it's "browser dependent." The point of making it a standard is so it wouldn't be browser dependent...

That said, I think there are definitely pros and cons for both specs. I'm kind of excited about the APIs though. (Also, the APIs are technically being developed by a third party, the Web APIs working group.)