Hacker News Reader

Comment by simonw | Hacker News Reader

> Team realizes their table wont scale, but their data is provided via API. They plan and execute the migration next sprint.

... followed by howls of anguish from the rest of the business when it turns out they were relying on reports generated from a data warehouse which incorporated a copy of that MySQL database and was being populated by an undocumented, not-in-version-control cron script running on a PC under a long-departed team member's desk.

(I'm not saying this is good, but it's not an unlikely scenario.)

View on HN

> they were relying on reports generated from a data warehouse which incorporated a copy of that MySQL database and was being populated by an undocumented, not-in-version-control cron script running on a PC under a long-departed team member's desk.

This definitely happens but at some point someone with authority needs to show technical leadership and say "you cannot do this no matter how desperately you need those reports." If you don't have anyone in your org who can do that, you're screwed regardless.

A lot of organizations are screwed.

I do agree with that. Microservices are not a good idea whatsoever for organizations with weak senior technical people. Which is probably 90%+ of businesses.

> ... followed by howls of anguish from the rest of the business when it turns out they were relying on reports generated from a data warehouse which incorporated a copy of that MySQL database and was being populated by an undocumented, not-in-version-control cron script running on a PC under a long-departed team member's desk.

Once you get to this point, there's no path forward. Either you have to making some breaking changes or your product is calcified at that point.

If this is a real concern then you should be asking what you can do to keep from getting into that state, and the answer is encapsulating services in defined interfaces/boundaries that are small enough that the team understands everything going on in the critical database layer.

This was why the "if you breach the service boundary you're fired" Amazon memo was issued.

another reason why you provide data only over API - don't reach into my tables and lock me into an implementation.

An approach I like better than "only access my data via API" is this:

The team that maintains the service is also responsible for how that service is represented in the data warehouse.

The data warehouse tables - effectively denormalized copies of the data that the service stores - are treated as another API contract - they are clearly documented and tested as such.

If the team refactors, they also update the scripts that populate the data warehouse.

If that results in specific columns etc becoming invalid they document that in their release notes, and ideally notify other affected teams.

This same thing can be applied to contracts when firing events, etc. I point people to https://engineering.linkedin.com/distributed-systems/log-wha... and use the same approach to ownership.

Yeah, having a documented stream of published events in Kafka is a similar API contract the team can be responsible for - it might even double as the channel through which the data warehouse is populated.

Reply on news.ycombinator.com