Amazon Athena: Query S3 Using SQL
aws.amazon.com
aws.amazon.com
If anyone from AWS is here: how is this used internally at Amazon?
I wonder if this is essentially a Presto SaaS product?
https://www.dropbox.com/s/s4cw5x7yyrdl3ch/Screenshot%202016-...
Even the pricing is same: $5 / TB of data scanned.
> Amazon Athena supports a wide variety of data formats like CSV, TSV, JSON, or Textfiles and also supports open source columnar formats such as Apache ORC and Apache Parquet. Athena also supports compressed data in Snappy, Zlib, and GZIP formats. By compressing, partitioning, and using columnar formats you can improve performance and reduce your costs.
Then again, both BigQuey and Snowflake require that you move data into their storage engine (Redshift too), and that's an additional step that's proportional to the size and complexity of your data. At the same time, it's stupid to store your logs as OLAP optimized formats and completely lose legibility. In sum, Athena trades off performance for convenience.
No matter what database vendors say, you can't defy the principles of computer science.
Apache Drill is a schema discovery on read approach that can handle some of this. Its not perfect, but it does simplify some of the process where its capabilities fit the task at hand.
https://cloud.google.com/bigquery/federated-data-sources
(I'm Felipe Hoffa and I work for Google https://twitter.com/felipehoffa)
Other formats are schema-less (JSON,CSV, etc.) or not supported by Redshift (ORC, Parquet). Perhaps less efficient for some queries (AVRO is not a columnar format) but still useful.
Athena is really interesting and if it can be as it is advertised "Serverless SQL" then they've got a killer product in the pipes: A future where developers no longer need to spend time on scaling, configuring, maintaining, strategizing deployments but upload code and instantly begin reaping the benefits of serverless tech.
The only missing component that would be a killer feature is something that answers to Azure's Active Directory. It would be nice if we had serverless plug-and-play user authentication and access control that integrated with Lambda and Athena.
I'd imagine some sort of "RoR on Serverless" type of framework that will scaffold out CRUD, User Management & REST Api is going to be in the works as well.
The only potential downside I see at the moment for Serverless is the uncertainty surrounding cold boots, it will directly affect user experience. It's fine when you got enough traffic to keep things in the "warm" state but there needs to be no dead zone when the call to the API Gateway is taking many seconds waiting for Lambda function to fire.
As for the cold boot issue, I thought the standing solution was to have a "fast-exit" ping-like code-path within the lambda. Query it on a regular basis (you can even do it with a lambda scheduled-event). That way your lambda should be kept warm.
That completely flew under my radar, not sure why I didn't see it before (oh that's right I was heads down in Azure).
With Athena the circle is complete for me.
That fast exit ping thing is pretty cool, any more information regarding that?
Your comment is probably the most valuable one I came across to date since signing up, I wish there was a way to award a gold star like on reddit :D
There's very little objection at this point in moving to a Serverless architecture = Athena (SQL) + Lambda (CPU) + Cognito (User).
You can hook your lambda up to a cloudwatch scheduled event (http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Sc...). If you don't do much when this happens (like, you exit immediately), then you won't be charged much compute time. Can't find the reference to how often you should ping it to keep it warm, probably varies, IIRC hourly would be plenty.
Off-loading this burden on a cloud vendor whose sales are directly tied to the security, gives me peace of mind. Ashley Madison had no financial incentive to keep their user base as secure as possible or actively defend against it using R&D. Yahoo Mail as well. But AWS and Active Directory are products that sell this security and for them to fuck up on an epic scale would mean their end.
so the tldr is piggybacking on the tremendous resources from a cloud giant which frees up resources to focus on the core product.
This is not to say that it's impossible to roll out your own wheel, I'm just saying it makes more sense to align the financial incentives of vendors maintaining the user base security. This may or may not mean I'm open to focusing on companies solely focused on user auth/management products, it's tough to beat the branding and trust built into AWS & Azure that developers are voting everyday with their money & data.
Athena does not gurantee for timing, use it for async call and offline processing
what did you mean by your last sentence?
I'm confused though. The monthly fees for dynamodb only apply after you exceed the free tier, and for someone who is unable to commit to a monthly fee because they envision low usage, shouldn't the free tier be sufficient? (Honest question, I'm looking at using dynamodb, but comments like this make me think I'm missing something)
Simply point to your data in Amazon S3
It's really only useful for a small list of file formats. Doesn't really do much for you if you primarily use s3 for binary data or static web hosting.