Uber's technology is reportedly 'hanging by a thread'
businessinsider.com
businessinsider.com
> Uber’s chief technology officer, Thuan Pham, later wrote to his staff that the mistake “reflects an amateurism with our overall engineering organization, its culture, its processes, and its operation.”
This makes it sound as if the two engineers in charge of the Node.js and Python systems are bickering over which technology stack is better and refuse to compromise. I get there is worry about career progression if your backend option loses, the glory of being the engineer in charge of the entire backend of Uber, and the different specs of different technologies. But to hold the entire company hostage seems like it would be a career-ending move to me, regardless of sides. Then again, I am not in management.
It would be, if you lose. If you win, you are rich beyond the dreams of avarice. It would also be a career ending move to admit that your system is less than the other system and everyone will remember you as that loser whose system was done better by the current VP of engineering.
Or redo the whole thing in Go.
The post I was replying to was asking why an individual lead engineer in this scenario would do this. They have reasons to fight and no reason to concede. Their bosses have every reason to, at minimum, get both sides to "hug it out" so why haven't they?
Is it such a bad thing to have many different languages in play as long as the SLAs are met?
It is faster and more scalable than Go plus it has a usable debugger.
career progression goes faster if you aren't limited by these sorts of choices. being able to learn something new fast and apply the same principles all over the shop is something that engineers do...
you work with what you are given.
Choosing X over Y could have meant they got out the door quicker than a competitor and got them to the point they're at now, even if Y makes more sense now at the scale they're at.
At the scale of Uber and given that there would be a lot of legacy hanging around, I would probably chose to build on top of the kind of scala libraries that twitter has been putting out (which it sounds like from the other article about uber's micro services currently on HN they are doing).
The serious statement behind my slightly tongue in cheek remark earlier, was that I don't think either python or node are suitable for building infrastructure type applications that will form the backbone of a constelation of SOA type applications. Round the edges, it's slightly more defensible. However for central infrastructure, I'd rather have rather more foot-bullet barriers than those offer.
If you're really serious you go C or C++, not interpreted languages at all. But that said, plenty of big scale stuff is running on PHP, Python and Ruby. The language matters less than the people using it and the architecture design.
In my experience Haskell also neatly address the other two points of people and architecture: the average [0] level of skill of developers in the Haskell community is higher, as is their average level of grit and determination (because of the learning curve). Haskell almost encourages constructing your applications as operations over streams of events (a la event sourcing, unified log etc), which for my money is the best way to think about designing most platform type apps right now.
[0] Note: I'm not claiming that all Haskell developers are coding ubermensch, just that the average level of the pool of talent is higher.
You could then let both sides continue to write the code in the language they are comfortable with but still work on a common platform. Plus being on the JVM it is fast and scalable.
It is not sensible to just stop what you are doing whilst you retrain and rehire your engineering team to learn a new language. Especially in a competitive hiring market like San Francisco.
1200? I'd be very interested to know what they're all doing. Not saying that what the company is doing is easy on that scale, but it's hard to see how throwing a thousand people at the problem can be an effective solution. Unless many of them are working on new products? But Uber seems pretty young to be investing that much in R&D.
For a fast-growing business, it would seem a huge win to not have to worry about physically scaling your infrastructure. And I can't imagine that their infrastructure size is so large (they're not, for example, indexing the internet) that they would get a huge cost savings from using their own hardware.
But certainly, I must be missing something?
Despite what Cloud providers want you to think, scaling (for most of us) is largely an architecture, design and software issue and it tends to be rather specific to your system (until you get to the point where you have to build your own). "Auto-scaling" doesn't help you make sure you aren't opening too many forking connections to your database, it doesn't eliminate coarse locks, un-optimized system calls, making sure you take advantage of cache locality, sharding, denormalization or really anything that requires effort.
The game changes a bit with the SaaS stuff that PaaS vendors are selling (dynamodb, for example), but then you pay a lock-in price.
That is not going to get the engineers on your side to solve the problem. Instead it will create even more friction between managers and engineers.
"Uber Raided By Dutch Authorities, Seen As 'Criminal Organization'" http://yro.slashdot.org/story/15/09/29/2328232/uber-raided-b...
That kind of problem seems remarkably common - the end of months and years can be peak times for businesses but it's also the kind of date that people pick for their project milestones ("we'll have the servers in by the end of the year").
edit:
Paywall. I get it.
Next I'd grid the area (maybe 250m2?) into blocks. When a user does a search, figure out his or her block, and start looking for cars in the current block, expanding outwards. You read-lock the blocks as you examine them. You only write lock 2 blocks as a car moves from one block to another (the blocks themselves would have a list of cars, maybe an array, maybe an rtree, maybe another grid).
It all falls apart when a single server can't handle all the load for a region. But then you could sub-divide the region, connect the servers with a queue (this car is now your responsibility), and let clients (not necessarily devices, but the api servers consuming these data servers) join the data.
The only case where I see 1 server failing is # of requests. All their drivers in the world probably fit in a few GB of memory (if that). 99% of a car's movement requires no locking (they stay in the same block), so there's very little write locking...I dunno...give it a 24 core server (or more)...
Maybe the problem is node and python. I don't know python runtime well enough, but this kind of setup is a nightmare for node. Sharing data across processes just isn't what it was meant to do (it rather fork and have a copy, but then your memory doubles per fork, and you have to keep it in sync). That's true for a lot of dynamic languages.
sqrt(34000) is nearly 200 miles on a side. And that doesn't really even cover all the areas that one might Uber from or to. So you'll have a lot of people crossing shard boundaries.
Greater LA has some 18mm residents, and again that doesn't count some of the places very close but not really IN the area. Places you might drive to in 15 minutes from the edge, over a pass.
But sharding comes at the cost of complexity.
I work on an uber-like service and we use crude sharding between countries.
The problem is:
* sometimes your sharding is too coarse - eg. two cities which have no cross traffic
* sometimes your sharding is too fine - eg. cross border traffic
But the real complexity is;
* managing configuration and logic differences between countries and cities
* hosting so many different sharded clusters, routing between them, and keeping them all updated.
And that's where all the work goes.