Visualize AWS Cloud Diagrams
aws.amazon.com
aws.amazon.com
I'm from AWS and my team are responsible for development of this project (AWS Perspective).
It's important to note that Perspective is not a traditional diagramming tool like draw.io or Visio. Perspective is intended to display resources that already exist rather than draw new ones.
To do this, Perspective inventories resources in your accounts (Lambdas, DBs, EC2 instances, ENIs, EBS volumes etc.) and can automatically create a diagrammatic representations of that data. Perspective also discovers the relationships between resources and will automatically expose them on the diagram.
The underlying data sits in a graph database (Amazon Neptune). This allows you to explore the relationships in the graph (diagram) on the fly visually to discover how your resources interact with one another.
We also have the early stages of a feature to show the estimated cost of the resources in the diagrams you create.
The project is open source and our roadmap is public. I encourage you to check it out and add any commentary for features you'd like to see (or bugs you find!).
https://github.com/awslabs/aws-perspective
I'm also happy to answer any questions.
Or expressed a bit differently: what are the infra + 2 data stores doing that's a major improvement over processing raw AWS Config + CUR data which should easily fit in laptop memory. We're talking megabytes of data for a point in time snapshot, right?
I'm assuming like most things AWS, this is not targeted at your small-sized startup running 2 VPCs, a handful of subnets, and 10-20 EC2 instances. This is likely built specifically for major enterprises running a hundred thousand instances, with hundreds of VPCs, thousands of subnets, hundreds of thousands of security groups, etc. In such environments, a static SVG isn't feasible and you might want something that allows more analytics instead, which I think is the entire point of using Neptune.
That's a good question.
Whilst Perspective sources a lot of data from AWS Config, it also supplements and enriches the data from other sources where resources are not supported or where we wanted to show more detail. We'll be adding in other data sources in future, too.
Additionally, the AWS Config query language is quite helpful but doesn't expose a graph of the resources. Our approach is to pull the individual relationship data out and represent those in a graph DB (Amazon Neptune). Backing on to a graph database gives us the ability to let users navigate through the relationships of interest on the fly, rather than dumping what would inevitably be a complicated mess on the screen if we just showed everything.
Using this approach is also one of the ways we've tried to solve the UI/UX challenges presented in this space. Rather than say "show me all the things" that would lead to a complex mess of interconnections (and probably explode browsers) we instead encourage a workflow that starts with a resource or workload and explores outwards from there. It also allows us to implement some interesting features in future -- we intend to keep working on the project.
The ElasticSearch cluster is used by the discovery process to locate resources based on the metadata we collect; it's part of the way we enrich the data and relationships beyond what we get from Config.
No, the diagrams of Perspective were not created with Perspective -- we had to stick to existing style guidelines.
If there are more you'd like to see added, please raise an issue: https://github.com/awslabs/aws-perspective
An eventing system that lets you hook node/edge events world be awesome as well.
Will take a look tonight, really interested in this.
Luckily it appears to be a "solution" instead of a core AWS product, otherwise I was about to say AWS has turned into a parody of itself.
It's much harder, I assume, to get security / teams to allow another team to crawl through its customer data (the Lambda team owns which customers have how many Lambdas running where, for example). Now imagine doing that for every AWS resource to form a graph of relationships. To get around that they deployed something that customers are wholly responsible for.
Forcing customers to cover the bill for something AWS should provide out of the box, in this subpar experience where they also need to manage all the infrastructure, is the opposite of customer obsession.
I doubt they were "getting around" anything. The customer being responsible for it is likely the entire point, because the customer who asked for this to be created probably specifically requested that (based on my experience working with AWS SAs).
This post is an "AWS Solutions Implementation". This isn't a service like S3/EC2/Lambda/etc created to be an integral part of AWS. A "Solutions Implementation" is basically the output of a customer making a very specific request from AWS Solutions Architects to build something customized (eg, customer doesn't want to use AWS Config but they want to build something from scratch with Lambda and RDS that works the same way), and then the solutions architects decide to share it publicly just in case any other customers find the same niche solution useful. It isn't meant for wide appeal or to be used by everyone.
I wouldn't be surprised if the fact that this post is getting decent attention means that AWS will use this solution as a proof of concept for investigating a future managed AWS service where the customer doesn't have to fully manage all of these moving pieces, but this post isn't that.
I doubt a customer asked AWS: hey, can I get a graph of how the resources in my accounts all interconnect while maintaining the clusters for the data stores, handling upgrade, security, etc. myself and paying hundreds of dollars a month for the pleasure?
The alternative is a customer asked for a way to see how resources were connected and how much it was costing them, and AWS, unable to deliver because of N hurdles (I mentioned security, but no doubt others), delivered this instead. Again: this should be a managed service and free.
I do consulting for a company that works with a lot of AWS customers and I think you'd be surprised. They don't put it in those exact terms, of course, but I have personal experience with a customer who specifically did want (I think due to their own internal regulatory standards) to maintain their own data stores rather than having an AWS managed service. In my example, it was using AWS Config versus having their own in-house software built on top of Lambda and RDS that did the same thing.
Another situation I've seen a lot is that really large companies have a tendency to think they are a super special snowflake with special use cases that no AWS-built service would ever fulfill, and thus they suffer from "not invented here syndrome". I've consulted many companies where I pitched them using an off-the-shelf product or service, but they will insist that it only covers 90% of their use cases and instead they will build their own custom solution using the AWS core services as building blocks, resulting in something similar to what is seen in the OP.
For the record, I think it's dumb as hell in most cases and I'm not in any way advocating that this solution in the OP is preferable in any way... but unfortunately I do understand how/why it was probably created.
1. Why do you need a graph database? You could almost certainly keep the entire thing in memory, who has >16GB of infrastructure?
2. 500+ dollars a month for a service that I'll use very infrequently?
3. I feel like all I actually want is something that takes CloudFormation and gives me a picture, for free, in a python script on my laptop. But ok, fine, what about things not in CF, and other metadata sure, but then see (1) and (2).
If this were, say, one dollar a month (or serverless) and the infrastructure was totally abstracted away I'd be a lot more excited.
edit: OK so (2) isn't really their fault, since this isn't an AWS service, but still. And 600 bucks isn't that bad for many companies, I just think there's a bit of a pricetag shock on this one.
you might find this interesting then:
https://github.com/aws-cloudformation/cfn-python-lint/pull/1...
https://github.com/aws-cloudformation/aws-cfn-lint-visual-st...
If this covers a reasonable percentage of AWS services it would be more than cost justified at $1-2k/mo for our environment.
But I looked at that cloud formation design and immediately thought “crap that’s probably going to cost way too much to run”. Anyone got a good up to date cloud formation to dollars per month tool and want to estimate it? (I’m on a mobile at the moment)
ES network traffic wouldn’t cost if you are only retrieving traffic in the same region.
You can find an estimated cost breakdown here: https://docs.aws.amazon.com/solutions/latest/aws-perspective...
(Disclosure: My team developed this project)
My strong suspicion is that this "AWS Perspective" is something that one of AWS's customers insisted on being built customized for their environment, and then the Solution Architects who designed it for the customer also wanted to get some "external credit" (AWS employees have to create a certain number of externally-facing blog posts/GitHub contributions/etc) so they uploaded it here.
Lucidchart is a great product but we were really turned-off & disappointed they essentially took a feature away from us.
It's free to run locally (doesn't use or require Config), the data is normalized & output as JSON for you to do whatever you want with it. It has better coverage than similar tools (>250 resources & growing), and provides relationship mapping between them as well. It's still a work in progress and might have rough edges, so feel free to file issues to help make it better.
[1] - https://github.com/trek10inc/awsets [2] - https://www.trek10.com/blog/awsets-aws-resource-listing-made...
Abridge collects data across your configured set of AWS accounts & regions. It doesn't draw architecture diagrams, but instead will give you x-vs-y visualizations for a number of different relationships .. IAM users vs. groups, Lambda functions vs. runtimes, EC2 instances vs. keypair, etc.
It'll also provide inventory tables / CSV dumps of the different resource types, and simple free search across all resources.
Security-wise, Abridge collects data via cross-account trusts with the SecurityAudit role, and expires all collected data after 48 hours - more at https://abridge.io/security/.
I kind of feel like you would need $20k+/month spend for it to be worth $500 a month to do this.