Hundreds of microservices”
Can people with expertise in this situation speak about how typical it is? I guess if you have all those services then you need your IDE to help - but surely this isn’t super common? If it is, what sort of size of micro service are we talking here? I’ve worked in orgs with 3M line codebases, and I’d struggle to find a hundred ways to partition the code naturally.
* Each team (let's call it eight engineers per team on average) typically managed a few services.
* Services were typically ~thousands of lines of code to ~tens of thousands. Rarely, there were very small services that were hundreds of lines of code. There was a giant service -- the original Rails app -- called "Monorail" that had millions of lines of code, but was actively being dismantled. My understanding is that there is now a code freeze on Monorail, and the only changes allowed are changes that remove code from it and move it to a smaller service (either an existing one or a new one).
* Services communicated with each other over HTTP or Kafka, typically using Thrift, although also sometimes using JSON. Airbnb had an in-house "service mesh" (developed before the term service mesh existed) that abstracted away individual machines and was capable of operating at the HTTP or TCP layer.
A few classic examples of mostly-single-responsibility services:
* Prometheus (which I wrote) abstracted away reverse geocoding and correctly caching and manipulating results from geocoder APIs.
* Medusa abstracted away ranking and filtering listings by date/amenities/etc.
* Micasa abstracted away basic CRUD operations on individual listings.
Airbnb intentionally used codenames for services rather than calling them by what they did, which I wasn't a huge fan of; I forget most of the codenames now (and hardly anyone knew all of the codenames offhand even at the time). Some other example without the cute names were the pricing recommendation service (used to recommend prices to new hosts), the calendar management service, the experience CRUD service, etc.
Data:
- Canonical data repositories for each of: order, transaction, customer, feedback, geography, privacy opt-in/out.
- Hadoop SQL facade.
- Support contact analytical service for a particular type of issue.
Business operations:
- Core order fulfillment service (orders populate into repository when complete).
- Support contact operational services: ingestion, routing, and auto-resolution.
Product infrastructure:
- Machine learning model serving engine.
- Map drawing service.
- Translations service.
- Dynamic configuration service.
- A/B test service.
- Server-Sent-Events hub service for our mobile apps.
- Facades for each of: voice, SMS, mobile native pushes, email.
- Marketing campaign service (dumber but more convenient facade to the previous).
Basic infrastructure:
- Column store.
- Secrets service.
- Cache service.
- Blob storage service.
- Workflow engine.
- Timer service (technically a subset of a workflow engine).
- Message queue.
Each one of these is backed by a team of 2-10 people. I have been inside 2 or 3 of them to fix bugs that affected me.
Microservices didn't become popular because the code could naturally be refactored that way, but because they enforced an interface boundary that worked better for mid to large sized organizations with multiple teams of engineers and levels of management where the communication overhead is much higher.
Each team has developed it's own way to work with it's subset of services - I doubt every team across every org could figure out how to run everyone elses domains to even make checking out the world a useful exercise.
As a long term corporate exercise, maybe that would "be awesome", but not trivial to do and PAINFUL to support and stay on top of over time.
We had a hard enough time keeping test suites up to date - I can't imagine trying to keep our portfolio "runnable" for everyone else.
I'm not sure whether it was really worth it for us, but it did solve that problem.