> I'm also not sure how to make the jump from exporting AWS_ACCESS_KEY_ID and having my instances automatically request the permissions they need - STS?

Check out instance profiles. This feature allows any AWS API-aware application to request credentials on demand, eliminating key management/rotation:

docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html