I didn't see the video. But given that a cab-service has a natural sharding point (i.e., per city), I don't get why scaling is such an issue.
EDIT: to downvoters: why shoot the messenger? :)
Is that strange? I've got 6 repos at work for various utilities and "personal" projects I'm working on in addition to the 4 repos for team wide projects.
Edit: I remembered more personal projects, make that almost 3 dozen personal Git repos.
Or maybe some instruction into how to assemble the organization.
10 git repositories per microservice
On-call shifts are going to be interesting with an average of 2.5 engineers per service, not to mention handling people switching teams or leaving the company.
They did that; the reason is job security.
But I can't imagine how you could possibly need 8000 git repositories unless you're massively over engineering your problem. Project structure tends to reflect the organizations that build them.
I've thought about this problem before, both for a related problem space and with friends working in this specific space. The short version is to create a grid where each square holds car data (id, status, type, x, y, ...) in-memory. Any write-lock of concern is only needed when a car changes grid, and then only on the two grids in questions. This can be layered multiple levels, and your final car-holding structure could be an r-tree or something.
The grid for a city can be sharded across multiple servers. And, if you told me that was necessary, fine..but as-is, I'm suspicious that a pretty basic server can't handle a tens of thousands of cars sending updates every second.
Friends tell me the heavy processing is in routing / map stuff, but this is relatively stateless and can be sent off to a pool of workers to handle.
The routing is very complex too but as you note scales well, until you want to start routing/pickups based on the realtime location of other cars.
- How do you distribute the car actors on nodes, assuming the number of nodes is variable? (I think riak_core looks interesting, but it does not seem to have a way to guarantee unicity of something - it's rather built to replicate the data on multiple nodes for redundancy)
- What happens if a node fails? What mechanism is going to respawn the car actors on a different node? How do you ensure minimal downtime for the cars involved?
- What happens if there's a netsplit, e.g. how do you ensure no split brain where two nodes think they're responsible for a car?
It feels to me like the traditional erlang-process-per-request architecture coupled with a distributed store (riak or w/e) makes it possible to avoid the very difficult problem of ensuring one and only one actor per car in a distributed environment.
I think you are right about the riak core stuff - you could probably keep track of cars using some sort of distributed hash and kill multiple cars if they were to ever spawn.
In fact a way of instantiating processes and finding them based on a CRDT is probably a pretty cool little project...
That describes no system, ever, including Erlang. TANSTAFL
Although... Ethernet latency would probably make it tough to stick to 2ms.
Analytics can run in batch mode. Also it is read-only operation. So it is far simpler than designing a distributed database.
As an aside, why would Uber want to reinvent a distributed database for storing user information?
The things you highlighted are very light load comparatively. Sign in, load account info, then dispatch to location specific shard.