Call me maybe: Aerospike
aphyr.com
aphyr.com
We have the enterprise edition of Aerospike, allowing us to be in constant contact with their support team and developers. A couple weeks later, and we still have no idea why this is happening. When dealing with monetary values, these fluctuations are very bad for us. Needless to say, we have begun migrating away from Aerospike.
I'm sure many of you know, this leads to quite a few issues.
The first I found when debugging an issue. I noticed the counters going up, dip down, then continuing up. Rinse repeat. (e.g., 158 -> 160 -> 158 -> 170 -> 175 -> 173 -> 180)
The second I found was when trying to debugging the previous issue. I noticed the counters were essentially cycling. (e.g., 158 -> 160 -> 170 -> 158 -> 160 -> 170). This just repeated for the duration we watched the counters (approximately five minutes).
Please note that I used small numbers here. The counters I was monitoring were in the hundred millions, and I saw decrements in average between 2-3k.
How big is this cluster? Are you writing and reading to the entire cluster, or do you have certain nodes that you write to and others that you read from?
We have two clusters of 8 nodes each. Each cluster is setup with 2 factor replication. The clusters are setup with cross datacenter replication.
Your read / write question is a little hard to answer. In Aerospike, a given key will always reside on the same node, something to do with how they optimize their storage. Which means that anytime you write to, or read from, a given key your query will always be routed to the same node.
The suggested method of solving this is to have two keys, one for each cluster, and XDR both keys. Then add them together in the app. You can maybe do it through a lua script, though I haven't tried.
For future consideration and delayed evaluation of course. I guess if you absolutely must have the most up-to-date information so you can make decisions on it RIGHT NOW that wouldn't work very well :|
Or would it? If your bids and stuff are also going through the event stream..
p.s. I am working in an Ad Network but not plugging into exchange. Our system is not capable for that.
For example, imagine an ad campaign spending $30k/month at a rate of $5 per 1,000 impressions. The customer may want their budget spread evenly throughout the month, so the software sets a daily budget of $1000. But this really represents 200,000 daily impressions, each of which is a transaction that subtracts from the available balance in real-time. The buyers software is talking to an ad exchange and keeping track of the budget every time an individual impression is won.
To add some more complexity, the impressions are probably billed as second-price auctions, so they aren't all exactly $0.005 each. Some are $0.00493, some are $0.00471, ect. Each one of these numbers is reported back from the exchange to the buyer's software in real time and the buyer is responsible for managing their budget.
This is just an example, but hopefully it illustrates how it can become impractical to account for this kind of thing using something more traditional like PostgreSQL. It would be reasonable to log all the impressions to something like Hadoop for the analytical piece of the software, but there needs to be something more real-time for budgeting to prevent overspending. The big ad exchanges can host hundreds of thousands or even millions of auctions per second, so not turning off bidding can be very costly.
This process of auctioning ad impressions across many buyers through an API is called real-time bidding.
So, fine, a probabilistic model might work well. And you might decide to bid on 100% of impressions. And you might decide that you have to bid $200 CPM to win -- which you're OK doing, because they're sparse.
And then say that FooConf happens in NYC and your aggressive $200 CPM bid 100% of the time blows out your budget.
Often you can charge the customer you're acting on behalf of your actual spend + X% up to their campaign threshold. So you really want to ensure that you spend as much as possible, without spending too much. Pacing is hard. Google AdWords, for example, only promise to hit your budget +/- 20% over a 1 month period.
With a real-time data stack you can avoid the duplicated ad a good percent of the time. Better experience for buyers, for publishers, and for users.
Yeah, but when that happens I usually don't think, oh hey they are lacking an optimal in memory distributed database solution.
I think, well... their engineers suck. Or they don't care. Pick one.
edit: His point is vague, so there is nothing technical to respond to. I am very much interested in a good technical example - but the things mentioned so far are by all appearances relatively straight-forward and linear, hence lack of effort or bad engineering are the only reasonable assumptions left.
Like most problems, scale makes it hard.
A cookie won't fit all that data and a more traditional database generally won't work. In memory k/v stores like Redis won't work due to data size (TBs of data). Hbase/Cassandra/etc sort of work with latency in the 5ms range. That's fairly expensive in a 90ms SLA, but you can make it work. It does limit the amount of work you are able to do.
During that delay you might have actually already processed new bid requests (auctions) for the same user.
Depending on the order's characteristics and how much you're willing to deviate from target - especially when observed within a small time window - the above poses additional challenges w.r.t. overspending.
I trust enough to get the job done, but not enough to get bitten when these systems drop data. Because here's the truth: they all drop data.
As is away from Aerospike to... ?
Thanks, very interesting anecdote/case.
Note that nothing is set in stone. The decision to begin migrations only happened today. It is possible that we will end up using some other technology altogether, or even we find out the issues we are having with Aerospike and continue using that service.
Point is, @aphyr skewers everybody. Aerospike is just the flavor of the month.
FoundationDB ran Jepsen internally and reported stuff[0], Kyle never worked with it.
[0] http://blog.foundationdb.com/call-me-maybe-foundationdb-vs-j... half broken now, none of the images load for me. @aphyr seems to have taken them at their word wrt testing though: https://twitter.com/aphyr/status/405017101804396546
We used it in production about 3-4 years ago and it was a nightmare from both usage and especially maintenance point. Fortunately we had a flat-files based backup system so we were able to rescue data every! Single! Time! the damn thing crashed and took (part of) data with it.
Of course, this is anecdotal evidence, and things might have changed from then, but I wouldn't touch it. Life is too short.
EDIT: Also, I am curious how the results in the above link would compare to aphyr's if he performed the test on HBase?
We use is in a very real time use case with latency requirements of single digit milliseconds, and if you tweak it the right way, you can the required performance from it, along with easy horizontal scaling.
Also, I am curious too for aphyr to take on HBase, but I don't think the result would be different since running Jepsen is straightfoward and not much to a person's interpretation. The results and further experiments are what aphyr does nicely.
OK, let me try to be more constructive. Since accounts are independent, shard based on account (in the application, not in some magic shard-distributing layer). Treat each shard as its own cluster.
If you want super fast requests but can accept being down for an hour or two a couple times a year, a shard can be a single beefy host with a replicating slave. I'd consider either Redis or Mysql/Postgresql. Really, these old-style sql databases can be the fastest things that have the kind of consistency you need.
I've maintained a mongodb cluster configured a couple of different ways. Performance at high load and reasonable consistency is not as great as some older alternatives.
We* tested the increment functionality heavily (300K-1M aggregate ops/sec) before we turned it on in revenue service. We use it for a couple of different things, event counting is absolutely the major use case.
In a single-cluster world, it works phenomenally well. In a XDR world, things get a little tricky, and we had to change the way our application logic worked to compensate for it.
Any more information you can share about your use case?
*a big ad tech company that uses Aerospike heavily
If performance is an issue, you can make the counters available in a second database that's only for reading, and updated from the original deltas.
Good read.
"We want to sponsor your distributed database research, but not your Barbie animated GIF production."
To me, it was immediately apparent that they are making the whole word Aerospike look like a rocket, a motif they repeat through their home-page.
Worker 2: "I'll go restart it."
See also: The frequent interchangeability of the words "system" and "computer".
I admire the work that Aphyr does - though at the end of the day, I need to build systems that work for the problem I'm trying to solve (and I have to choose from real things that are available).
Aerospike isn't the solution to every storage problem, and if you are choosing technology based on marketing material, you're probably going to be disappointed.
These technologies in general are trying to address really hard problems and design and architecture is the art of balancing tradeoffs. Nothing is going to be perfect. Yet.
These seem like highly detailed Github Issues (infact the recent elasticsearch was a GH issue-turned-blog post), and these issue are brought to attention so that they could be fixed - not to slander the name of the company (and when they are, everyone benefits). IIRC, even after finding these bugs were published he continued to use elasticsearch.
Given how hard these problems are and how difficult they can be to reproduce, these writeups seem to be the most appropriate way to highlight these issues.
That said, if I was an aerospike user I'd be happier knowing this issue exists, someone has debugged it, and supported a detailed report about rather than being called in at 3am and discovering our data is funky.
Those blog posts are also a great at debunking marketing claims.
Isn't it kind of annoying that some technical projects bolster their popularity/ecosystem with very fancy websites and impressive/competitive claims, but to really do your job right you have to throw all that away? The best you can do is try to get a sense from the reports of others who have tried something (and may or may not have been rigorous in their evaluation) so you can pick good candidates to even put through trials. (so again, thanks Aphyr)
I came across Aerospike technology via a pre-existing system at a previous employer, and watched that system scale up and perform in a serious way. It wasn't all unicorns and roses all the time as real life never is, but in the context of the real world, it was great. The software is rock solid in a way I've rarely come across, and support was spectacular. (I forget my current production clusters are even running sometimes they are so stable, reliable and self-operating)
And at the end of the day, there was no other solution out there remotely competitive that we could find. And I looked - not because we were dissatisfied, but because that was our fiduciary responsibility to the company, to ensure that we were deploying the most cost-effective systems that met our feature and performance requirements.
Ultimately yes - I think that as an engineer, you need to understand what your tools are really capable of and avoid doing what I call "BDD" (Blog-Driven Design). That isn't the ideal answer - it would be nice to have a reliable understanding of the capabilities of the materials we use to build systems (like civil engineers can reason about materials like steel and concrete in repeatable ways) but what we call "software engineering and architecture" is still a very young discipline, very often with unrealistic expectations about our ability to deliver in given budgetary and temporal constraints, so we do what we can.
Something Aerospike didn't realize.
(And nope, sorry, I'm completely uninterested in your anecdotes about how you haven't personally lost data when [1] there's a clear data loss scenario highlighted in the post, [2] Aerospike actively recommend services like EC2 and GCE that routinely partition, and [3] there are people in this thread who have experienced the same problems).
</explainer>
All of other databases were tested in clustered mode. Why not PostgreSQL as well ?
It's also worth posing this question in reverse: what would happen if these distributed databases were tested in a single-cluster configuration? As noted in the most recent article on Elasticsearch, many of them (e.g. Elasticsearch, Cassandra, and Riak) acknowledge writes before fsync and can therefore lose data due to issues like `kill -9`, power loss, and other exceptional conditions, while Postgres doesn't. For a single-node database this robustness is very important, while he argues that it isn't as important for a distributed one. Because these databases aren't designed to be used as single nodes, aphyr didn't substantially ding them for that. Again, what's important is whether the database does what its documentation says it does when used as its documentation says it should be used.
Jepson tests network partitions... so less useful.
Of course, I would love if someone gave me the mandate to go out and build something like F1...
And honestly, if you have money, it's a lot simpler and less risky to just hire 20 DBAs and programmers to build a database application that can handle that kind of operation. Low latency, network-partition-resistant, high-performance database applications are not a new thing.
No, Aphyr said the data loss is ok for ad tracking and analytics because it doesn't matter. That's very different.
And if that's the case, then why make those claims.. they could just as easily give accurate info to their customers, and the customer could decide if that fits their case. Instead they claim something very difficult (if not impossible), and let their customers find out it's not true (possibly after it's too late, and they've already lost valuable data).
I'm constantly surprised by the general tone of comments on posts like these as if it's some crazy revelation that this software still obeys the fundamental laws of distributed systems.
There is no perfect database out there, all of them will fail with network partitions. Aerospike was designed to work in clusters that are very close together, often the same rack. It has much tighter timings and tolerances in exchange for providing much higher performance in certain situations and definitely has one of the best SSD focused storage systems I've come across.
If you don't have a high performance network interconnect between nodes, then there will be more issues with Aerospike since it relies on that more than some other system that use Paxos for all writes (like aphyr mentions). We run several TB's of data accessed at 100k+ TPS including very fine grained counters and everything works. And yes, we run on the cloud in AWS and SoftLayer and have yet to have major problems with the proper network setup.
Btw, there is a comment below from the current CTO of AppNexus, one of the companies that pioneered real-time bidding for digital ads and runs several million auctions per second on one of the biggest ad exchanges available. They were the first customer for Aerospike and from everything I've learned from their team, it works really well for them, and they definitely are not happy to just "lose" data however insignificant it might seem. Volume changes everything and even a fraction of a percent will add up. We trust Aerospike because it's been hardened by lots of much much larger companies with very high production usage, the key is being aware of all the technical requirements and the environment you're deploying in.
I think the real major issue here that people seem upset with are the general claims and marketing information. I can't speak to all that and there are definitely some things like 100% uptime which do seem overly confident, but this is true of every single technology vendor out there unfortunately. I'm not saying Aerospike is any better or worse as a company but marketing material only goes so far and it would surprise me if further research wasn't done for any mission critical system.
Some of them will fail in a way that keeps your data safe, others will fail in a way that preserves uptime but gives you temporarily inconsistent data. Aerospike apparently does neither. Why is it unreasonable to expect them not to falsely claim otherwise?
The "crazy revelation" for me was not that Aerospike's software is, like everything else, subject to the CAP theorem. It's that they apparently think it's awesome to claim that it isn't, and charge tens of thousands of dollars for their product on that justification.
2) Yes, marketing claims are BS. If this was a reason to not use something, we'd have to stop using pretty much every other commercial piece of software we have. That's why we test and run software in our environment, and there... aerospike works. Really well. Even with network partitions. So I can understand kyle's tests in this post and the reasoning and results but there's still a big gap between this testing and the reality our company has experienced.
Which is what I at least have indeed opted to do; I avoid commercial software like the plague for this very reason, using it only when there isn't an alternative (like when it's a legacy system that has to be interfaced with). There are plenty of free software projects that don't make outrageous marketing claims and - therefore - aren't nearly as susceptible to disappointment and wasted money.
Aerospike's claims border on the realm of false advertising (if they don't actually classify as false advertising, which is a big "if"; the claim of 100% uptime is dubious at best and more likely to be an outright-malicious lie). Why should they get my money?
What bothers me more than anything was Aerospike felt the need to compare a benchmark done on an in-memory dataset directly with a Cassandra benchmark on dataset that was many fold larger into disk. They made this comparison and said "look! Aerospike is x times faster than Cassandra!"
That was the end of giving a shit for me. When people feel they need to lie to convince you of something, that is when I know that I don't want what they want.
Incidentally, that is also how I stopped caring about politics.
Why would you trust something after its untrustworthiness is demonstrated before your eyes? Just because some other companies use it and haven't yet publicized dissatisfaction? That is not how you make sound engineering decisions.
We use this 24/7 in a production system and have not encountered any issues and it matches actual data and experience from real conversations and meetings with other companies. We don't make decisions from blog posts.
Consider that if you have not experienced any issues yet it might say more about your network stability than aerospike. Of course you can chose to ignore it, but network partitions eventually do happen, and when it does I hope your data is not mission critical.
We have network partitions all the time, that's how we upgrade. On average each node is replaced every 2 weeks and we just terminate it through the API (both softlayer and aws). No big deal and we haven't lost any data yet, confirmed by other records in other datastores that have to match up.
If aphyr's post is the ultimate rating, why would anyone use anything thing else he's written about?
Of course not, software is about trade-offs, and every company has different use cases. Is it their primary datastore? if they lose data do they lose some data samples for a recommendation engine? or someone's money? I wouldn't assume "if it works for them it works for me".
> Trying to solve hardware issues with software is bound to lead to misery.
I strongly disagree with that, I believe exactly the oposite.
> If aphyr's post is the ultimate rating, why would anyone use anything thing else he's written about?
He does not judge the system's usefulness, throughput, etc. But he's a good benchmark for distributed system's reliability. While he might not test every possible scenario, if he says software X loses data on conditions Y, I do believe him. It's still up to me to decide if that matters for my use case or not.
eg: aerospike will have counters to cap a certain transaction and when we do offline aggregations from logs written through a completely different system, the numbers have to match.
If something is capped to spend $100 and the aggregations don't match up to exactly $100, then there's something wrong, especially with very fine grain numbers.