Since Serilog does structured logging, I can use either an ElasticSearch or Mongo sink and do complex queries.
If I routinely need to log into an instance to troubleshoot, I need to be capturing data and sending it to a central logging system.
It means you can connect to it by just knowing its instance ID.
Adding the IP address everywhere also works.
There can be some nice SSH config options though, like using a particular key for everything *.prod.myaws.com
I haven't had to manage SSH keys in a long time ;)
With this I just have a bash function for my various environments (e.g. dev = dssm) where I provide in the instance ID giving me issues if I really need to log into the server.
e.g.
function dssm { aws --region us-west-2 --profile my-dev-profile-name ssm start-session --target $1 }
Then:
dssm i-abcdef123456
And I'm dropped into a shell. SSM Session manager is far from perfect, but it gets the job done, and is fully auditable, gets logged (including commands ran), and best of all works with SAML IAM profiles right out the gate. No more sharing keys, no more managing keys, it's great!
> Access: When troubleshooting, we save time not having to look up the instance’s internal IP address for SSH access.
SSH access is absolutely a last resort, but can be necessary in certain cases (like when Filebeat breaks...). Turning SSH off completely (i.e. "No SSH") is certainly better for security and something we may pursue.
I mentioned in another comment here that SSH is just one example, we can also easily hit endpoints with curl via hostname.
Also mentioned in the post are other tools (like Grafana dashboards) have an expectation of unique hostnames.
I suppose it's possible to build enough logging to account for an interactive SSH session for debugging problems...but that would be massive.
I ran out of disk space. Why?
That’s why you need a central logging facility. If you’re using AWS, you could store your structured JSON logs in S3 and query them with Athena. (https://medium.com/quiq-blog/store-json-logs-on-s3-for-searc...)
Of course there are other ways both using AWS and third party services. Centralized logging is a solved problem.
AWS isn’t going to run out of disk space any time soon. You could also use a lifecycle policy to delete old logs or move them to a lower cost storage depending on your retention policy.
I’m not saying that I have never had to log on to a VM to troubleshoot, but that’s a sign of the need of better logging.
And if my logging infrastructure isn’t good, how pray tell will I troubleshoot my programs running on Lambda or Fargate?
> how pray tell will I troubleshoot my programs running on Lambda or Fargate?
That is indeed a big problem running on Lambda and Fargate.
In my experience, Fargate isn't very commonly used and Lambda is used for only relatively simple things.
It’s the same concept. If you’re troubleshooting at any point involves needing to log in to an EC2 instance, you might as well have a few bespoke servers called “Web01” and “Web02”. You’re just using ASG to create pets at scale. We run an ASG in production that scales from 2 to 30 instances based on the number of messages in a queue, lambdas running all of the time, some a Fargate tasks etc. it would be a nightmare to troubleshoot all of those processes without centralized, queryable logs.
In my experience, Fargate isn't very commonly used and Lambda is used for only relatively simple things.
And that experience is representative of the entire AWS ecosystem?
For one part we had a legacy service needing to connect to the services in the ASG and the best way to implement it was with round-robin DNS. So the lambda would update a DNS record contianing all the ASG host ips.
Also, because we had some had some semi stateful legacy instances that where basically lift and shift to AWS, but I wanted to have them in ASG to keep our environment similar until we could refactor them into real cattle.