iamsomewalrus

I have personally only used them for debugging one off questions. That was my original intent. We do have teams that are considering using the snapshots for ML problems.

We use carrier pigeons from an S3 data center high in the Himalayas to a CloudFront distribution center in Atlantic City (don't ask me, ask the pigeons) to serve all requests.

Just BI.

Absolutely! We have internships in San Francisco and Seattle. Usually, they coincide with the summer, but we on occasion have Fall interns.

Reach out to me feeneyj @ amazon if you'd like I can forward you to the right people.

Having been back in SQL land for a bit (vanilla MySQL on RDS) I have to say that I _love_ a well designed SQL database. I forgot how much I had given up in NoSQL land.

Goodreads hit scaling issues a while ago with Active Record and a single database so we broke up the data into separate MySQL servers. At that point joining data across DB servers is impossible so we went with Redshift for BI. Nowadays we would probably go with a datalake on S3.

I did! I'm a big fan of the Last Week in AWS newsletter.

https://twitter.com/iamsomewalrus/status/1016368717880389632

:this: A concept that's underlying the move to a datalake architecture (read: keeping your data in its rawest form, and its transforms in S3 or HDFS) is decoupling your compute from storage.

Motivating example: you have huge tables in Redshift that are either infrequently accessed or the usefulness of the data decays over time (website logs, customer order information). In this scenario you're paying a lot just to keep data in Redshift (storage) but a large subset of the data is laying dormant (no compute).

If you're bought into the Redshift ecosystem this is where Redshift Spectrum comes in. If you're a smaller company you could just store the data in S3 and "spin up" the compute when you need it (Athena, Glue jobs, or Elastic Map Reduce clusters).

That’s good to hear! The underlying architecture project has been in motion for a few years. We’re excited that it’s out.

No, but we have a really responsive customer service team that watches the Goodreads feedback group very closely.

Come join my team and I’ll be happy to spill all the beans!

We can, and will do better in the areas you’ve mentioned. I’ve been bitten by that same search bug :-(. I’ll cut a ticket for that.

That’s “the question”. I will do my best to bluff through it.

Engineering team size: you have teams large enough (~3-4) to own and iterate on a subset of related functionality for the long term.

Tooling: you have a builder tools type team that provides a tooling and observability happy path.

Traffic scale: you have functionality that operates at 2 or more order of magnitude higher that the rest of your application.

Decoupled: you have functionality that can be decoupled from your main app and, most importantly, isn’t required for your main app’s uptime. Like a search service or something.

A combination of the above.

Data will eventually end up there. Our product and business teams are heavy users of Redshift.

As mentioned in another comment I’ve found having Dynamo snapshots in Athena really useful as an oncall to sanity check snapshots (what was the state of Harry Potter 3 months ago compared to now?) and to answer product questions that can only be answered from the raw production data.

We have another GR team that does that and it works well. It’s complicated by the fact that at Amazon every team uses their own AWS account. Concretely, a service’s DynamoDB tables don’t exist in the same AWS account let alone the same VPC as the redshift cluster. Obviously, you can figure out the permissions, etc.

We’re trying to get to a place where we have the data in S3 for engineers to build products off of and for the oncalls to do sanity checks and the data in Redshift for our BI needs.

Thanks all. Long time lurker, first time commenter.

There’s no real policy that I’m aware of internally for teams to use microservices. Amazon has a lot of tooling to make it easy to spin up services, however.

The first major project after being acquired was to make a pared down Goodreads experience available on the Kindle Paperwhite. Our first services came out of that initiative to provide a buffer between the Kindle traffic and the Goodreads Rails app.

That being said I’ll be the first to caution small teams should avoid microservices at first for fear of creating a distributed monolith.

Athena is just a front end for the data that a typical user can understand (SQL!). the real value is that the data is: in S3, in a more efficient format (parquet), and available in the Glue catalog.

The other replies got it right w.r.t. other BI tools. If you’re using Tableau I think it integrates with Redshift, right? In that case Redshift Spectrum is an option.

If you don’t have any existing BI tools then Quicksight is an option or alternatively you can spin up an Elastic Map Reduce (EMR) cluster with your fav open source BI tools

Did we just become best friends with our “I am the Walrus” references?

The DB scrape uses a template from Data Pipeline that under the hood uses the Dynamo DB Scan API. Not really surprising, but that API uses JSON. I wanted to use as much off the shelf software as I could to get data into Athena.

In this architecture Lambda is only used to listen to the SNS topic that fires when the Data Pipeline job succeeds or fails so we’re not pushing the limits of Lambda at all. You’d probably hit an EC2 limit Witt Data Pipeline before hitting your Lambda limit on the account.

Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.

My team has run into the same problem in a large Rails app. We decided on a combination of service objects and data access objects (DAO) where appropriate. In a lot of cases this has really helped with the grep-ability of our codebase.

Service objects allow us to encapsulate complex operations that touch several domains and tables through a simple API. They're just plain objects.

DAOs fulfill a specific case of providing a simple CRUD API to a domain. We use a lot of DynamoDB so the DAO allows us to hide the complexity of reaching out to several tables in order to return a result. We also make sure that the DAO returns an immutable object that can't reach back into the datastore like an ActiveRecord object can (#save! methods come to mind).

The difference between a service object and a DAO is a little blurry but is enforced through quick design discussion and code reviews.

I have two invites. Email in profile.