Curious he does not mention the hosted elasticsearch+kibana on Amazon. Even integrates with cloudwatch and lambda to pull in the data.
For basic use cases it might be fine, but AWS ES doesn't support X Pack, lags behind official releases, and I even recall it being more expensive once you setup production level ES domains since master nodes are paid for separately. XPack is huge as far as solving headaches with ES, even for the authentication management alone.
We use Terraform to provision our stack though and it seems that the Elastic Cloud doesn't have any programmatic configuration options. How does your team orchestrate your elasticsearch cluster(s)?
EDIT: I should clarify, our use case is not ELK, it's search over a modest data set.
We're pretty write heavy/low read with a comparatively small data set (~50GB expected over 4 months). I know ES can be tuned for that use case to improve index throughput at the cost of slower searching but I never tried setting it up on AES (I'm guessing it'd go in the CF-Template).
One challenge with elastic.co is there's no equivalent to Amazons ElasticsearchDestinationFirehose. We were putting data into a Firehose which batched, backed up to S3 and accounted for failures via a Lambda but elastic.co is giving you "raw" ES instances, so you need to handle those usecases.
You could probably write a Kinesis stream that replicated that functionality(and from what I understand Firehose came to be because that's exactly what people were doing), but we control the devices uploading to the Lambda, so we had the devices act as the buffer (increased send thresholds, and propagated failures to send to ES to our devices which retry automatically) and sent a duplicate record to an S3DestinationFirehose for backup and replaying data
I'm currently tasked with getting a SaaS application onto Amazon and picking it up as I go along, mostly just from the docs. It's been very hard to find high-level overviews of ways to use AWS's ops tools like CloudFormation, most articles I've found are about one specific API and not about architecturing AWS apps.
We run serverless (which in turn applies the CF templates) on Jenkins instead of locally (which also helps automate passing the correct stage and parameters). I guess the equivalent without serverless would be just be using aws-cli, since serverless doesn't do anything special to the CF templates
There was a great webinar on production deployments with SAM (which you could replace with CF) via CodePipeline, but I can't seem to find the slides for it anywhere. If we "did it again" we'd probably use CodePipline/CodeBuild(We just used Jenkins because we didn't really have a chance to look at other options)