DivineTraube

The JMeter load tests were conducted to test the throughput of the CDN and backend setup (browser caching was disabled). The load generating servers where hosted in Frankfurt (Germany) where the CDN (Fastly) also has an edge location. This lead to average response time of 5ms. We experienced some hickups within JMeter causing single requests to stall up to 5 seconds. For the 10 million requests we got an 99.9th percentile of below 10ms. We couldn't quite figure out why JMeter had these hiccups, maybe those were GC pauses, maybe something else, but it was consistent across different setups and servers.

And it not only hurts because of revenue. Slow sites also let users preceive quality and credibility as lower (Fogg et al. 2001; Bouch, Kuchinsky, and Bhatti 2000). The startup/brand will end up being seen as less interesting and attractive (Ramsay, Barbesi, and Preece 1998; Skadberg and Kimmel 2004). So being slow or offline can even have a long-lasting negative impact on the coolness of your startup.

HTTP has a good solution/proposal for this: the server can include a stale-on-error=someTimeInSeconds header in addition to the TTL and then every cache is allowed to continue serving stale data for the specified time while the origin is unreachable. Probably a good idea to include such a mechanism in DNS, too.

https://tools.ietf.org/html/rfc5861

So AMP does two major things:

(1) It prescribes a stripped-down version of HTML and uses a JS loader to render fast and load as much resources as possible asynchronously

(2) It caches the website in the Google CDN and delivers it via HTTP/2

By applying best practices of structuring web applications and optimizing for the critical rendering path (1) can be achieved, too. The AMP loader is just an opinionated way to do this for simple pages. Everyone can get similar results without AMP by following web performance best practices like [1]:

- Reducing the critical resources needed

- Reducing the critical bytes which must be transferred

- Loading JS, CSS and HTML templates asynchronously

- Rendering the page progressively

- Minifying & Concatenating CSS, JS and images

And there is lots of good tooling for this (e.g. postcss, processhtml, cssmin, UglifyJS, imagemin, critical, gulp-rev-all, ...)

(2) is not only harder. It is what limits the broad applicability of AMP. Cached data in the Google CDN cannot be invalidated, neither in the CDN itself nor in ISP interception caches, corporate proxies or the browser cache [2]. The effective consequence of this is that you can only use AMP when the content is mostly static. This is the case for news websites or other publications that are only changed by human editors. It completely breaks down when you try to create a dynamic site, for example a social network or a shop.

That is why our startup Baqend [3] takes a different route. We say that developers are clever enough to use existing tooling to achieve (1): an efficient rendering and loading experience. And for (2) we add a caching scheme that also employs CDNs for delivery but keeps data consistent. This is made possible by a simple process:

1. When a browser connects to a Baqend-based website, it loads a Bloom filter containing all potentially stale cached URLs.

2. Every stale URL is requested using HTTP revalidation to refresh stale copies and update caches.

3. When an update operation changes a resource (e.g. an image, JSON object or even a complex query result) its URL is instantly invalidated in the CDN [4] and marked as stale in the Bloom filter.

4. When loading resource, a statistical estimation of the expected TTL (cache liftime) is made. Whenever that prediction fails, the Bloom filter and the automatic CDN invalidation compensate the difference between estimation and real lifetime.

Using this scheme (developed at the University of Hamburg in cooperation with Baqend), every kind of dynamic data can be treated as cachable data and the applicability of is not limited to data that seldomly changes and even works for write-heavy resources with rich consistency guarantees (Δ-Atomicity, Read-Your-Writes, Monotonic Reads, Monotonic Writes, Causal Consistency) [5].

Of course I'm biased but you'd like to see AMP-like acceleration coupled with fresh cached data plus tooling, layout and frameworks of your choice, have a look at our Backend-as-a-Service.

[1] https://developers.google.com/web/fundamentals/performance/c....

[2] https://github.com/ampproject/amphtml/issues/1901

[3] http://www.baqend.com/

[4] https://www.fastly.com/blog/building-fast-and-reliable-purgi...

[5] http://www.slideshare.net/felixgessert/talk-cache-sketches-u...

We have not seen any serious DDoS attacks, apart from the ones we created through our own heavy load testing. UDP attacks are not really a problem, since the servers are only accessible by Fastly over TCP using a combination of SSL authentication and IP whitelisting.

It's pronounced "Backend" ;-)

And since we are in the Backend-as-a-Service market, the name is not all that unfitting. Although it cannot be denied that from time to time some people think we are French an spelled "Baquend".

We (Baqend) use an approach that is somewhat different from what has been proposed here so far:

- Every one of our servers rate limits critical resources, i.e. the ones that cannot be cached. The servers autoscale when neccessary.

- As rate limiting is expensive (you have to remember every IP/resource pair across all servers) we keep that state in a locally approximated representation using a ring buffer of Bloom filters.

- Every cacheable resource is cached in our CDN (Fastly) with TTLs estimated via an exponential decay model over past reads and writes.

- When a user exceeds his rate limit the IP is temporarily banned at the CDN-level. This is achieved through custom Varnish VCLs deployed in Fastly. Essentially the logic relies on the bakend returning a 429 Too Many Requests for a particular URL that is then cached using the requester's ID as a hash key. Using the restart mechanism of Varnish's state machine, this can be done without any performance penalty for normal requests. The duration of the ban simply is the TTL.

TL;DR: Every abusive request is detected at the backend servers using approximations via Bloom filters and then a temporary ban is cached in the CDN for that IP.

Felix, author of the article and founder of Baqend [1] here. I'm happy to answer any questions. If you have suggestions for the next version of the article, we're eager to hear them.

[1] http://www.baqend.com

I'm also very curious about CouchDB 2.0 and whether CouchDB will be able to make a comeback. When I talked to Adam Kocoloski about a year ago, I got the impression that it takes a very good technical approach (consistent hashing, causal hash histories, MongoDB queries, etc.). However, the implementation is now mainly driven by Cloudant and CouchDB kept alive by them. I hope they manage to rekindle the old fascination they once had.

In any case I will be happy to update the article and include it.

I was oversimplifying of course, Redis Cluster definitely deserves a place in the comparison. We are actually very satisfied with using Redis in production but had to work around the PubSub limitations using keyspace notifications, which is why I'm biased regarding that point.

In any case, it's difficult in terms of presentation, since a "normal" master-slave-replicated Redis is a totally different system from Redis Cluster as the distribution models of Redis Cluster also affects the functional properties to a large extend, e.g. regarding atomic Multi/Lua blocks and all types of multi-key operations.

The failover solutions for relational database systems are a good point, they too should be included.

Regarding use cases, I totally agree: every system should be discussed in the light of the use cases it tries to solve. I personally think that Redis Cluster with a choice for trading consistency against latency would open up a whole new range of use cases that are currently not a good fit for Redis Cluster. For example, we have a Redis-Coordinator project (not open source, yet) which behaves similar to a scalable Zookeeper (BTW also neither CA nor CP [1]). However, it has weaker guarantees, due to lack of tuning knobs in Redis and Redis Cluster regarding consistency.

[1] https://martin.kleppmann.com/2015/05/11/please-stop-calling-...

I agree, Redis Cluster has been available for a while. From a practical perspective we did not deem it production-ready yet, due to various shortcomings, e.g.: -Redis Cluter being neither AP nor CP with rather unsatisfying justifications [1] -Strong scalability issues, for instance with PubSub [2]

But you are right, I think Redis Cluster could be added as a separate system.

[1] https://aphyr.com/posts/283-jepsen-redis [2] https://github.com/antirez/redis/issues/2672

We tried our best to incorporate these results in the cases where they reflect fundamental shortcomings not bugs (of which Aphyr found quite a few). But you are right, we could include a list of popular examples, where description and experimental findings diverge. Which cases did you have in mind? MongoDB being CP?

The central message of the tutorial - the NoSQL decision tree - is available as a PDF cheat sheet too: http://www.baqend.com/files/nosqltree.pdf

There also is a paper on the NoSQL classification scheme used in the slides: http://www.baqend.com/files/nosql-survey.pdf

CEO from Baqend here.

This announcement will have a huge impact for the mobile dev community that relies on Backend-as-a-Service systems. I personally think that Facebook had several reasons to shut down Parse:

1) The technology stack was really fragmented and often rewritten in large parts. I still have this statistic in mind how their 200 Rails API servers were only able to serve 15 requests/s each [1]. If you look at the database technology inside Facebook, there is much superior infrastructure that was never really integrated into Parse (Haystack@OSDI'10, Tao@ATC'13, F4@OSDI'14, Extended Apache Giraph@VLDB'15, RocksDB, etc.)

2) Parse did not have a core competitive advantage: it was just the first company to whole-heartedly pick up the BaaS-paradigm with sufficient man power and a good understanding of developers' needs. The technology itself was not particularly innovative in any way, just (mostly) solid engineering. However, there remained really basic limitations that were never addressed [2]. For instance the only (!) way to safely handle concurrency control was through counter data types.

3) The model of Parse promotes independent apps and websites outside the Facebook universe.

4) The pricing in increments of guaranteed 30 requests/s was okay for simple apps but absolutely useless for anything beyond that. In particular for websites which as of 2016 do an average of 100 requests per page load [2] a single user can leave a Parse app rate-limited or down.

The main asset of Parse were their great client SDKs and well-written documentation.

This is why we made a plan: we will fork the Parse SDKs to offer seamless continuation of apps relying on them, including Push and the other features dropped in the open-source Parse Server. We opted for this approach as the Parse Server implementation on Github looks really brittle and the convoluted Parse REST API is really not an option. By doing this we hope to provide a scalable and long-term solution for developers looking to continue their Parse-based apps.

Baqend [2] is a pre-seed startup founded out of the database research group at the University of Hamburg (Germany). Our product launching into production within the next months uses a new approach to consistent web-caching reducing latency in common web workloads by up to an order of magnitude [5]. It is due to this background that we very eager to not only provide great usability (which Parse also did) but also acknowledge the need for complex data processing: low latency access, partial updates, continuous queries and ACID transactions.

We'll post a detailed plan on our blog, soon.

[1] http://blog.parse.com/learn/how-we-moved-our-api-from-ruby-t... [2] http://profi.co/all-the-limits-of-parse/ [3] http://httparchive.org/trends.php/ [4] http://www.baqend.com/ [5] http://www.btw-2015.de/res/proceedings/Hauptband/Wiss/Gesser...

This is the transaction protocol from Google Percolator - with slightly different wording. I don't know if that's a well known fact but I was surprised they did not even mention it. OSDI Paper on Percolator: https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...

Percolator is the processing system Google uses to manage its search index incrementally. It's built on BigTable which provides cell-level linearizability, similar to what CockroachDB seems to achieve by using Raft.

What CockroachDB calls "intents" are per-value columns called "write" in Percolator and ordered by BigTables timestamp system. Percolators "master lock" is rebranded to "switch" and, tada, there you have CockroachDBs fancy lockless algorithm.

Edit: Amazon DynamoDB, too, uses a very similar approach to this in their client-driven transaction implementation https://github.com/awslabs/dynamodb-transactions/blob/master...

It would be interesting to have one of the CochroachDB guys comment on the novelty of their approach.

If you have any other crash snippets, I'll be happy to include them.

While this looks promising, why not make Disque part of Redis by introducing the "message queue" data structure? This would allow to build more powerful, application-specific messaging abstractions, for instance a compare-and-swap-key-with-reliable-broadcast (CASKWREB in Redis terminology).