Amazon Aurora Backtrack
aws.amazon.com
aws.amazon.com
It's good to know that Aurora will try. It's not like it needs to be reliable or anything.
And, really, all of this is just servers in a datacenter, so you can have however many 9's of reliability, and still be That Guy that wins the fail lottery.
"Reliable" isn't a binary concept. There's no service that is 100% reliable.
Having a solution that works 99.99% of the time (which is "only" four nines) is probably good enough that people are willing to use it as a last-resort. (From what it seems, it's included for free, so it's not like this costs you anything anyway - at worst, you end up where you would be if they hadn't released this).
1. Writes to fail and your production service goes down because you couldn't guarantee Backtrack.
2. Your service stays up and you lose some amount of emergency undo ability.
https://www.allthingsdistributed.com/files/p1041-verbitski.p...
And the docs
There are actually two backup/restore methods.
1) Point In Time - streamed to S3. Guaranteed restore to a new cluster with less than 5 minutes data loss
2) Aurora Backtrack - On cluster buffer, up to 24 hours retention (no guarantee). "Instant" restore to the same cluster
So you should feel safe using the service (up to the 5 minute worst case scenario).
Obviously no one would ever want to use either backup method. However previously disaster recovery would involve spinning up a new cluster, deploying application changes with the new endpoint (or in a best case repointing a dns entry), waiting for the cache to warm up again. Now it becomes a 1 click fix through the UI with no application changes.
I only wish that every new feature didn't inevitably come with the caveat that it's only for the MySQL flavour of Aurora.
I understand both the engineering and product development reasons for doing so (different stack and MySQL is undoubtedly a much larger customer base), but it always makes these announcements a little underwhelming as an Aurora Postgres user.
Not only MySQL, but only MySQL version 5.6, which was released in 2013 and superceded by 5.7 in 2015. Aurora finally released support for 5.7 in February this year after years of development, but all new features so far have been stuck on the 5.6 branch which hasn't been very reassuring.
That being said, I love AWS, am Pro-Certified, and work with it everyday.
I know Oracle is a giant mean bully company, but at least their arrogance was never of the “world-destabilizing” kind like Facebook.
EDIT: changed rollback query to flashback query (flashback query can be used both to view or to actually change the DB)
https://docs.oracle.com/cd/E11882_01/backup.112/e10642/rcmfl...
> In Aurora, we have chosen a design point of tolerating (a) losing an entire AZ and one additional node (AZ+1) without losing data, and (b) losing an entire AZ without impacting the ability to write data. We achieve this by replicating each data item 6 ways across 3 AZs with 2 copies of each item in each AZ. We use a quorum model with 6 votes (V = 6), a write quorum of 4/6 (V w = 4), and a read quorum of 3/6 (V r = 3). With such a model, we can (a) lose a single AZ and one additional node (a failure of 3 nodes) without losing read availability, and (b) lose any two nodes, including a single AZ failure and maintain write availability. Ensuring read quorum enables us to rebuild write quorum by adding additional replica copies.
There are many 2 AZ regions in AWS, of course. I don't think you can stripe 3 copies per AZ, an AZ failure drops you to potentially 2/6, and if you allow for 2/6 and 3/6 writing you could have a split brain. Any thoughts how they manage that?
[1] https://www.allthingsdistributed.com/files/p1041-verbitski.p...
> Amazon S3 Standard, S3 Standard-Infrequent Access, and Amazon Glacier storage classes replicate data across a minimum of three AZs to protect against the loss of one entire AZ. This remains true in Regions where fewer than three AZs are publicly available.
Italic emphasis is mine.
They are slowly adding a 3rd AZ to regions that didn't have one already.
E.g.
https://aws.amazon.com/blogs/aws/now-open-third-aws-availabi... https://aws.amazon.com/about-aws/whats-new/2018/03/aws-govcl... http://www.datacenterdynamics.com/content-tracks/colo-cloud/...
Realistically I'd never use this feature because of the risk of data loss. I'd restore a new instance from backups and copy the lost data back over manually.
Sounds like a nightmare as far as relationships are concerned.
As another commenter pointed out, per-table would be scary for referential-integrity. And per-db kinda makes sense, but if you've got totally different use-cases hosted by the same MySQL you may be using the service incorrectly.
> I'd restore a new instance from backups and copy the lost data back over manually.
That's always been possible, but even with the best instances and most iops it could take hours to do that. So it's really designed for a different scenario.
I might be wrong but I believe Aurora backups can be restored quite a bit quicker than that, right? We are evaluating Aurora at work, hence my interest.
edit: And I'm being down voted because? Because HN does not enjoy the truth?
So, yes, they save snapshots and take you back to the time of your choosing based off that.
How do you know they use the snapshots method, rather than reversing the changes?
Okay, but you're most likely wrong with that. If you read about how Aurora works (https://www.allthingsdistributed.com/files/p1041-verbitski.p...) it seems likely that they will be using log entries to rollback database state, rather than retaining "snapshots".
SQS: There's AWS MQ, which is based on ActiveMQ and supports AMQP. If you're going with a more CNCF-focused stack and want to use NATS, I'm not aware of any hosted options.
You would just want to stay away from Aurora specific features like the one in OP.
Replicating it on a different platform could also be fine with a combination of logical and physical backups.
I think it's important to have a good framework for making these decisions. As you mentioned, EC2 and S3 are on one end of the spectrum: low switching costs and coupling, but also low in terms of differentiation at this point. Whereas other AWS services that are more bleeding edge could actually be the foundation of a temporary competitive advantage for your product, but today would introduce deep coupling of your service with AWS.
The trick is basically landing somewhere where you've made good bets: the ideal scenario is that you have a good chance of having an escape plan if/when the situation warrants it, while not missing out on the opportunity to take advantage of the services that your competitors (like, for example, those who are on the "EC2/S3 only" methodology you mention) may be forced to build themselves out of fear.
Unfortunately the best way to do this is to be able to predict the future: which services on AWS have high value and high longevity, and will have similar, cheap offerings on their competitors and/or via OSS in the coming years? S3 is an example of a good bet: if you adopted S3 when it first arrived, it may have felt like unnecessary couping with AWS, but it was a huge accelerant if your business relied upon large, reliable file storage. (In fact many startups likely today exist because they were able to bootstrap off of S3.) S3 was a good, long term bet -- now, you can drop in an OSS replacement that is API-compatible in a worst-case scenario. Other services on AWS have come-and-gone through their hayday without going through that transition, because they were either (in retrospect) transitional technology or basically "the wrong thing." So, it comes down to predicting :)
A couple of things that increase probability an AWS service is more destined for long term success + commoditzation (in my view):
- Built on open protocols/standards
- Simplicity in APIs and feature set
- Versatility of use-cases
- Deeper in the stack, vs user-facing
- No existing open source alternative with traction
- Backchannel confirmation that Amazon itself is using it :)
For example, DynamoDB when it first came out was a huge "no" for me personally -- it was a very quirky design, with complex, bespoke concepts around indexing, with lots of good open source projects offering alternatives, and I heard Amazon didn't trust it internally :) Whereas other services like Redshift ticked all the right boxes out of the gate. So there's some hope of making good bets here, but sometimes you do need to throw caution to the wind if something extremely compelling comes along that feels like it could be a shot-in-the-arm to your product despite not having most of these attributes.
Also, FWIW since I like calling my shots: I think the current iteration of AWS server-less tech (while a necessary first step towards understanding the design space, and a technical marvel imho) is probably not a good long term bet as a foundational technology for a product you would like to de-risk in this aspect, but we'll see :)
Take your Redshift example. It's an outstanding service but once you build apps around it and load a bunch of data the switching costs become very high. AWS can raise the price quite a bit before it makes economic sense for working apps to move off it.
Consequently a key question to ask is when do my favorite AWS services become cash cows? At that point your economics will change substantially. You may be living day-to-day in your business but the stock market thinks Amazon will extract large profits out of AWS in spite of growing competition. Food for thought...
Of course, in the "long term" (whatever that means), the market and shareholders expect to reap vast profits, and price that, along with the company's free cash flows, into the stock.
However, my personal thesis is that Amazon specifically is a bit of an anomaly in this regard: it seems unlikely to me that we will see any strategic shift in pricing from basically any Amazon-owned entity in our lifetimes -- the culture is too ingrained towards a strategy of complete and utter dominance over all markets, margin reduction being a key tool in the toolbox. So if I were to bet I would say pricing power is pretty low on the list of concerns of AWS lock in -- more concerning are things like unexpected conflicts of interest as Amazon eats the world (see also: Netflix), operational dependency (ultimately, you rely upon AWS ops' competency for your uptime, security, etc), end-of-lifing of services you depend on, opportunity cost vs other vendors who may offer better services, and other unknown unknowns.
That said, Amazon Prime prices just went up 20%, so Amazon will clearly raise prices if they think the market will bear it.
Moreover, "complete and utter dominance" sounds like a monopoly. It's difficult to think of a monopoly that didn't raise prices and/or decrease service once they were established. I would not start a business on Amazon without assessing the risk of tying my fate to a single large vendor, for cost as well as other reasons you cited like Amazon deciding to compete with you.
That's a bad example. Redshift is "wire compatible" with Postgres. You use the same drivers. There are a few popular extensions they added to Sql to copy files directly from and to S3 but that's about it.
Assuming you seek wire compatibility, your choice would therefore be a PostgreSQL-compliant DW, of which there are several. However, they have greatly differing cost and operational profiles, which make switching non-trivial. They also tend to support different versions of PG depending on when they forked from PostgreSQL.
I'm not saying it is a good idea to leave Redshift in some distant future where you might save a few pennies. Heck, I hardly ever say it's a good idea to rearchitect a core part of your infrastructure without having a very good reason -- saving a few dollars isn't one. But from the software side, you aren't stuck with Redshift and you aren't embedding a lot of proprietary AWS specific drivers in your code base.
You can say the same thing about a loan shark or high interest loan. Just because you can doesn’t mean you should.
I'd add that there's often a danger in the middle ground. Complexity and meaningful switching costs, without the benefit of interop, is likely the worst-case scenario. IOW, might be better to go all-in than muddle through with half-measures trying to avoid lock-in.
I've used both in production, with all sorts of advanced features, and (to my surprise) I've never yet found a difference.
I would have a hard time leaving Aurora though. Not because of normal "vendor lock-in", but because of the operational worry-free scaling, backups, replicas, point-in-time restores etc. that it offers. Basically, I'm "stuck" on Aurora just because no one else offers MySQL/PostgreSQL databases like that.
Given the ten-ish years I've been using AWS everything has gotten progressively cheaper and more feature rich. I see no signs of that stopping.
I'll happily take on the risk of that theoretical future wherein for some reason I need to unwind from AWS, and in turn continue to save countless hours of my and my engineers' time not spent tackling solved problems, and focusing on our core business instead.
I encourage my competitors to marry themselves to a vendor’s proprietary stack.
Source: 18 years in tech. Have done my share of unwinding from vendors
If the provider of a proprietary solution stops offering it you might have no easy path of migration where as hypothetically FOSS solutions are easier to self host or find a new hoster for (and thus reducing the cost of a potential migration)
But that goes back to the old saying "no one ever got fired for buying IBM." The same can be applied to Microsoft or AWS. Yes MS has put plenty of technologies in maintenance mode but you didn't have to rush to migrate. IBM still sells system that are compatible with what they sold in the 80s.
I would be more worried about my one colocated data center getting shut down or getting destroyed than the entire global AWS infrastructure that has data centers in 50+ availability zones.
As far as software, most AWS services are just managed versions of available software. Even if you're using lambdas for API endpoints, if you treat your lambdas like Controller actions, it shouldn't be that much of a heavy lift to concert them into standard MVC APIs equivalent.
For another perspective, listen to the podcast interview of Adrian Cockcroft, the guy who led the migration of Netflix's infrastructure from self hosted to AWS hosted.
http://www.se-radio.net/2014/12/episode-216-adrian-cockcroft...
On a more advanced level, using CloudFormation or Terraform is not rocket science and neither is setting up a CodePipeline.
When you grow big enough to need someone to handle netops full time, you can hire an overpriced consulting company or find a developer who knows the ins and outs of AWS and dev ops. But for smaller companies, managing AWS is not worthy of being a full time job if you know your stuff around automation.
I'm mostly either a "Senior Developer" or an "Architect" depending on how the wind is blowing, but from working with "AWS architects" and consulting companies, I know I can hold my own against most non-Netflix level AWS architects.
But I'm glad we have a third party consulting company to handle the drudgery that I don't want to do even though I do have admin access to the console.
What you're talking about here is survivorship bias -- yes, indeed you will find people paying big sums to migrate off of AWS for whatever reason. But they're still around! Unless of course you assume using AWS (or any vendor for that matter) provides no competitive advantage whatsoever, even in the short term.
The specific category of risk you are talking about here are tech ventures that fail because of a specific sequence of events:
- They choose AWS
- They, miraculously, create a successful business on AWS
- Some existential risk emerges due to their use of AWS
- They either fail or cannot afford to move off of AWS
- This singular decision causes their business to fail catastrophically
- And, they would have been successful had they not chosen AWS in the first place
This is a very specific path to failure and is highly unlikely relative to the more common cases of "they never found product-market fit" or "they had a toxic culture" or "they decided to build everything themselves and ran out of runway."
I work for a ~billion dollar organization in risk management. If developers can't certify to our team that you can pull your app off of $cloud_provider and have it running on prem or in another cloud provider in a few weeks, it doesn't go to a cloud provider. We'll assist with their IaaS/PaaS tooling selection, infrastructure orchestration, container lifecycle process, and other ancillary needs to ensure this requirement can be met.
The issue isn't survivorship bias, it's risk management, business continuity, and cost management. It's entirely possible your organization doesn't place a high dollar value on these risks, but that's a business decision each organization needs to make for itself as a whole, and its products and services individually.
A) I start a company, get some investors and sell the company to a larger company that has the resources to either keep maintaining it or change out the infrastructure (Instagram). Result: I profit
B) I start a company, gain product market-fit, become successful and I have the resources to gradually move my infrastructure when the need arises: Result - profit.
C) I worry about "vendor lock-in" from day one, spend a lot of time and resources on my own infrastructure and run out of money and never survive long enough for A or B.
I know which one I'm going to choose.
And as far as not "needing the complexity", the entire purpose of going to AWS is to avoid the complexity and up front cost of providing your own infrastructure.
The HN darling Dropbox started out on AWS because it was faster and then when they grew they built there own infrastructure.
Netflix built their own infrastructure and decided that infrastructure wasn't their core competency and went entirely to AWS.
There is no one size fits all.
The reality is you'll need to do it when you are tight on resources and you're stuck rebuilding what has become a very complex infrastructure (because you spent the last 5 years hooking into every proprietary bell and whistle) from scratch - with no internal expertise because you didn't start early when it was still a tractable problem.
I'm not saying you need to choose between AWS or running on your own physical hardware. But I am saying you don't want to lock in to the proprietary parts of AWS, which do tend to add lots of complexity to your infrastructure because they make that complexity easy to use.
Spinning up some ubuntu ec2 servers is fine - you can move those anywhere easily.
I've worked for a company that had two issues - badly written software and a badly architected AWS environment. Guess which one was easier to fix?
All of the companies that tied their fate to Webforms, Ruby (Twitter), or some other deprecated tech have far bigger problems than someone who tied themselves to AWS.
As far as AWS what complexity is that?
Databases - all of their databases are compatible with standard databases except for DynamoDB. DynamoDB sucks anyway. If I wanted a managed NoSQL database, I would go with one from Mongo.
Messaging - You usually end up sending and reviewing messages from one central place in your code, changing your messages system out is pretty simple.
Caching - ElastiCache exposes either a Memcached or RedisCache client.
Load balancing - the only thing special you do as far as software is have a health check endpoint. I used the same endpoint that I used on prem with Consul+Fabio.
ElasticSearch -same as using your own.
Autoscaling - I've got nothing. I've never tried doing autoscaling on prem.
Codepipeline/CodeBuild/Code Deploy - the triggering event is nothing more than a standard Web hook from your git repo of choice, Code Build is a preconfigured Docker Container that executes commands based on a yml file. I use a Python Lambda function for deployment, but I could run that in anything.
Even Lambda if written correctly - your handler should be skinny and treated like a Controller - shouldn't be that hard to port over.
Fargate/ECS - orchestration for standard Docker containers.
Parameter Store - yeah I could (and have) set up a cluster of Consul/Vault servers but why would I instead of using a service that ties in security roles and encryption with everything else?
Load Balancing - I've also set up my own Consul+Fabio clients on web servers/app servers for load balancing and service discovery. Not trying to do that anymore either when I can configure an ALB and an autoscaling group.
The software wouldn't be the hard part to move over. Even if we ran everything in EC2 instances, the hard part would be duplicating multiple geographically redundant VPCs in multiple locations in the US and an environment in Asia that was close to the outsourced developers.
Netflix went all in on AWS and much of their tooling is tied tightly into AWS. If Netflix can trust its entire business to AWS, I think anyone can.
This is a terrible argument. Netflix is simply a media streaming company, with very little importance compared to other businesses. Most of its data is served from its own CDN!
Check out a few of Netflix's blog posts, AWS re-invent presentations (available on YouTube), the podcast interview, I posted earlier, and thier open source project. Netflix uses a lot of AWS services.
Edit: just realized I duplicated scarface74’s comment. My bad!
The most likely outcome has already played out. A lot of people aren't moving out of their datacenters, they're just putting new workloads in AWS. Sure, some big players actually moved, like Netflix, because the gains outweighed the cost of moving, but most enterprises don't have the budget or don't care.
If something ever comes along that is significantly better, most people won't move out. They'll just put new workloads in the new thing.
Drew Houston, CEO of Dropbox, confirms my stance argued in this thread on Dropbox’s first earnings call: https://www.cnbc.com/2018/05/10/dropbox-earnings-q1-2018.htm...
Netflix moved out of their datacenter and into the cloud because their workload was different enough to warrant the investment. Most enterprises don't have enough of a different workload for that to make sense, so they don't move out of their datacenter.
Dropbox didn't actually move out of AWS, they just started storing new data in their new datacenters. They did it because it was cheaper at their scale, it had nothing to do with lock-in.
And for the same reasons, when the next big thing comes along, most companies won't move out of AWS. Unless there is a significant cost savings or significant new architecture, they won't move. They will just put new stuff in the new thing. Lock-in isn't an issue for most enterprises today, and won't be in the future. Inertia is the biggest thing that stops them from moving old workloads to AWS.
If you decided to build your stack on top of MS SQL Server in 1990 you would have still had a relatively painless upgrade path over the years.
We can also add products byy Oracle and Adobe.
The right approach would be to open an text file and start writing all your hardware drivers, an operating system, a database system, application services etc. so that you'll never ever be locked into anything, and you'll always be 100% in control of everything at all times.
Shit, at that point you'll also need to make sure you fabricate your own chips, lay your own pipes, invent your own transfer protocol, etc. as that's the only way to be absolutely certain you won't get locked in.
But then at that point, you've created a prison of your own demise.
I’ve seen it happen time and time again. Companies write their own frameworks, build systems, hardware provisioning systems, analytics systems, you name it... all to avoid “lock in”.
All of that home brew stuff usually sucks. Almost all of it becomes abandonware that one or two people in the company know how to change. Why? Cause all that stuff has nothing to do with how the business delivers value. All of it should have been provided by third party packages.
But now the company got big and they are locked into shitty homebrew garbage that would take a massive political and engineering effort to get out of.
Moral: it is just as easy to lock yourself into your own garbage as it is to lock yourself into a third party. Leave stuff that doesn’t add value to your business to people whose business it is to build that stuff. Beware of doing everything in-house.
The other solution is to use common components (standards or oss) and then you have nobody to rattle your saber at when it breaks.
But two of those options are for the most part self-inflicted by the developers. The other one is usually inflicted by management (vendor lock-in).
It's important to ask questions like that – "what happens if this service got away / becomes unaffordable / removes features we depend on". There's risks in that sense with using proprietary technology. Sometimes it'll be worth it, because the benefits of allowing you to deal more quickly with business problems will outweigh the technical risk. But I've definitely been in more than a few situations where a vendor has shut down a product or service that I depended on, resulting in a difficult or time-consuming migration.
Some of this risk is ameliorated by using open or popular standards and systems. Intel's not likely to stop producing x86 chips; GitHub is probably going to continue supporting git; Linux isn't going anywhere anytime soon. It's worth thinking about before using a service that nobody else offers!
This seems to be the middle ground. Making well-reasoned decisions about which vendors will have a long and well-maintained life of service at the expense of it being trendy or cool.
Meanwhile, the original post you replied to did not say to completely avoid AWS, only to use "standard EC2/S3 services" rather than provider-specific ones. It may not be the middle point you prefer, but it's hardly an extremist position worthy of ridicule.
It's important to provide a detailed analysis weighing the pros vs cons of building an application on any brand new service like GAE etc.
For instance, one con could've been "if they ever increase their prices by X amount, then we're fucked" would've been a pretty relevant con to consider before jumping into using it.
AWS may have contributed some new technologies, but for the most part, they've just wrapped things up in a GUI. I'm not denigrating that as a valid market niche to fill, but people should know that there are non-AWS solutions for virtually everything that AWS provides, and that your company's admins probably already employ a good portion of them.
A good question is, how is it different. With PIT restores, they create a new instance from snapshot and play back the logs up to the relevant point. With this Aurora-only feature, they take advantage of Aurora's storage capabilities, and rewind the same database to a previous point. No new instance, no cold recovery from snapshot, no long log replay.
This is a non-issue for the vast majority of products. You do your eval and you chose what makes sense. Short of hiking prices by 1000% you'll never move anyway (this also applied to building ridiculous abstractions because "maybe we'll want to switch cloud providers someday!" No, you won't.)
Also, one that was a licensed software that needed to support DB2, Oracle and MS-SQL to integrate with various state's agencies. The former was a huge pain, the latter was horrible to write code for.
For the most part, you use what works for you and stick with it. The more risky a platform, the less you rely on features like stored procedures, etc. In the end, I'd rather have vendor lock-in than have to write complex queries three times, and other really cumbersome abstractions, slowing things down a lot! Extrapolating to services beyond the DB.
In the end, the db migration was far less time/effort than maintaining an untethered software.
I wonder if we will be able to recognize Amazon in ten years.
RDS is fine too as long as one sticks to standard backends.
In general, if you're moving to AWS and just setting up some EC2 instances and using S3 in the fear of one day in the future it may not be what you want, you are wasting money. There are much cheaper pure hosting solutions. The benefits of using AWS is all of the services they provide that keep you from doing the "undifferentiated heavy lifting".
Don't be fooled, the compatibility is there so that it's easy to port applications to Aurora, not so it's easy to take them out.
Amazon still offers MySQL RDS. If you move to the much-more-expensive Aurora, it's generally because your MySQL RDS application is struggling, and Amazon says "Aurora knows special tricks to make the engine faster."
Some management headaches, like adjusting for disk growth, are handled by Aurora transparently, which is nice, but for the most part, I saw workloads going to Aurora because they were too slow on standard MySQL.
That means that instead of optimizing or fixing your application, you're relying on a single vendor's Secret Go Juice to make your application usable/performant. In such cases, unless the mainline distribution catches up in the meantime, you're going to be stuck because your stuff will be too slow on other stacks.
From the risk management POV, Aurora is at best a temporary solution while the program is refactored/optimized to be performant on standard distributions.
The second maxim is "no one ever got fired for buying IBM". If you're going to bet on a horse, there is always a chance that things go sideways. You might as well bet on the fastest horse with the best record.
If you had a choice between IBM, DEC VAX, and Stratus VOS back in the day and you went the safe route to go with IBM, you would have been vindicated two decades later. IBM is still selling compatible systems.
I could go through the trouble of staying "vendor neutral" and use tools like Consul+Vault with an AWS backend, Nomad, and Terraform (been there done that) but the time I'm wasting being vendor neutral, I can spin up equivalent services on AWS that cost less and I don't have to maintain them.
Most of the things that I depend on AWS for now, I was doing the equivalent of on prem before and after AWS existed. I'm glad to not have to do that anymore and just being able to press a few buttons.
And on top of that, in the realm of database engineering, Amazon is absolutely a newcomer, they've got another good 15 years before they can even start to be in contention for "reliable and trustworthy database vendor". Goes doubly when you consider that Aurora is essentially just a bunch of performance hacks to (old versions of) InnoDB and Postgres. Presumably, upstreams like MariaDB and PostgreSQL have forgone similar enhancements for a better reason than "only Amazon employees are smart enough to figure it out".
They did hire a bunch of experienced engineers. I'd rather have them actually contribute to postgres (which amazon pretty much doesn't do, some bug reports aside).
> Presumably, upstreams like MariaDB and PostgreSQL have forgone similar enhancements for a better reason than "only Amazon employees are smart enough to figure it out".
Some of them are harder to do if you don't have as much control over the environment as amazon does...
That's why no announcements cover change fundamental change to either database - all the features we're seeing a the kinds you'd do on a personal scale with SSD RAIDs and ZFS (or other copy on write FSes), except on a datacenter scale.
Have you seen the code see to know that they are just “performance hacks”? Have you thought that when you can actually spend millions on a project you might be able to throw some top notch engineers at it?
I think I used AWS for email octopus and maybe 1 other thing.
Idk how I've avoided AWS for so long.
There are plenty of companies, large and small, running very successfully on all these platforms; sabre rattling notwithstanding.
And like all technical debt, eventually you may find good reasons to rewrite, redesign, rethink, in order to achieve some different goals. But you might not have been around long enough to reach that stage if you hadn't taken on the debt early on.
So then is it worth it spending momentum (dollars + thought) on that stuff, or on solving the business problem?
And, really, if you're doing that from the beginning, you might find you save a lot of money by taking advantage of other providers' strengths.
Edit: Main difference seems to be new cluster vs. in place.
Edit: I guess it's also nice that it's in place. Some people probably don't have the flexibility to easily point applications at a new cluster in a repeatable way, so that may help folks prevent down time.
delete from ImportantTable where id = 1234
... and forgot to select the second line in some query editor before running.
delete from ImportantTable where 1 = 1;
TBH, I'm not sure how useful it is compared to the normal PITR. My usual guidance on recovering after a database fuck up is "put the app into maintenance mode and the database in read-only, do your investigation, run PITR, port user activity to the backup, do spot checks to determine everything got through, roll the app back to before the change, take the old database down, point the app at the new database, bring the system back online."
I'm sure a manager is thinking, "oh, we can be back online in 5 minutes, amazing!" But just because something broke doesn't mean users stopped hitting your database! You can't just hit Undo and throw all that out!
All this makes me really really wanna use Aurora. :)
I suppose it works fine on its own for small to medium databases, and it's a fantastic product. I just wish it was a bit better. :)
As for S3, WAL-E is probably what you wanted to use. Barman was more intended for on premises installation.
What if you had an e-commerce site? Other customers were placing orders, right? Credit cards being charged? You didn't catch your mistake instantly. So you have a window of maybe, 5 minutes or a hour in which other things happened. You can't simply forget those other transactions and throw away the data.
1) Point in time back ups can be hours old.
2) More importantly, “backups” are useless, it’s the “restores” that are valuable. And very few organizations have a well practiced muscle memory for restoring from a backup.
A turn key restore solution, with a per second granularity can both significantly decrease the loss window, and recovery time. Hope nobody gets to use it, but when you have, it can be a difference between a big and a small outage.
Per second granularity gives you ability to restore as close to the fatal mistake (or poison pill transaction) as possible.
After that, recovering transactions lost by the rewind can be really tricky (but not impossible). If the entire database got dropped, for example, it’s possible for the unwound writes to contain conflicting IDs.
[1] https://about.gitlab.com/2017/02/10/postmortem-of-database-o...
Why would someone want to rollback their own production database instead of PITR to a new database and switching over to it? Surely you would end up losing data because you wouldn't be able to reconcile the new data written to it.
Have we though? This could be one of those safety nets that makes me worse not better.
Aurora is a log structured db, they can just reset their DB to a particular LSN. i.e. doing "git reset --hard", which's O(1).