I wish they didn't hire so irresponsibly back in 2015 and 2016.
I wish they didn't hire so irresponsibly back in 2015 and 2016.
Can confirm this. It was a very poorly managed org because it was extremely grassroots driven—leadership was underempowered to say no to front line teams or hold groups accountable.
The overlaps and poorly considered projects described above are not an exaggeration. Many of them were designed for building the lead engineer's open source brand, not for company needs. Half of the projects described above have since been canceled.
Google:
- Hyperbahn
- Cherami
- Schemaless
- Peloton
- uDeploy
- m3db
- piper
- XYS
Contrary to their glossy open source and tech blog presentations, these projects were highly contentious internally and widely viewed as inferior to industry counterparts. Each of these projects had 5-20 engineers working on them for over a year; many of them are being EOLed or phased out currently. And this is just the list that I can publicly talk about. This development came at extraordinary cost for the org and, in the case of the people who were laid off, cost to people's careers.
Seems like you probably got a lot of shares and were lucky. Now that they IPO'd, do you consider that you came out ahead financially (compared to working elsewhere)?
That said with dilution, it's likely 2015 share price is less than it is today.
I think they moved to RSU's in mid 2015 or 16.
I'm especially curious about Schemaless, since it seemed like at one point every company tried to build a NoSQL KV store on top of sharded MySQL, realized how janky it is, and are currently decommissioning it.
The development team is pretty active. They've been doing an important effort to gain community adoption, e.g. persistency with MySQL, Postgre in the works, Prometheus style metrics, a proposal process or design docs. Very interesting.
I do agree there were definitely projects that were overly ambitious and optimistic that likely that should have been better planned. I think you are weighted firmly at one end of the spectrum however. I have some obvious bias being involved in one of the projects you outlined, but I still think you are blindly covering a swathe of projects in one go without any nuance at all.
The project page states "M3, a metrics platform, and M3DB, a distributed time series database, were developed at Uber out of necessity. After using what was available as open source and finding we were unable to use them at our scale due to issues with their reliability, cost and operationally intensive nature we built our own metrics platform piece by piece."[1]
Do teams just work on random projects like the above while other people fix bugs? Is it like a million mini startups internally all pitching an idea to the CEO all the time?
Or do you need 90% of people focused on how to develop your product at first, then suddenly it flips and you need 90% of your engineers to balance enormous amounts of network traffic and data storage?
Many of the projects are just a response to some other internal problem, like "the teams aren't using a shared platform", so they build a platform for the platforms. Persistent problems get ignored because nobody wants to rock the boat. The joys of working in a large corporation...
This is a great way to have engineers burn time on projects they think are interesting rather than projects that will actually have good ROI.
Larger companies with substantial profits can hire engineers explicitly to come up with POCs one after another.
I've seen a lot of vanity projects over the years, and often it feels like this, or a form of in-house retirement. That guy has done so much for us that now we just let him do whatever as long as HR is okay with it.
That said, Schemaless isn't going anywhere. Cassandra doesn't cover all the use-cases.
I fail to see how.
Considering Uber is reinventing the wheel a lot, the people who were laid off where probably working on some interesting problems. Uber is a fairly well known brand. Unless they are incredibly inapt at selling themselves, their career should do fine.
You're only in a bad bargaining position if you need a job to pay your bills.
The key in the discussion with the recruiter is not to say that you are unemployed but to say that you're taking time for yourself and looking for opportunities that meet your professional desires and baseline financial expectations. The key is to make it clear that you have no financial needs, just financial expectations.
Be prepared to use the Noel Smith-Wenkle salary negotiation method.
I took six months off after a large liquidity event and then went back into the job market. I don't think I lost any negotiating power - the trick was to make sure I times the interviews so that I'd have several offers at the same time and then could use the fact that company X was offering a certain amount to ensure I got the job at the company I liked the most.
I do just want to say one thing: most tech investors now think we are in a bubble. A downturn is likely at some point. Now would be a very good time to ensure you have six months of expenses in a savings account so you don't have to take the first job you get offered - I had to do that in 2008 after the financial crisis hit and my startup died and it wasn't a fun time.
There is a similar effect people have when they reach a certain level of financial independence - not only are offers better, but negotiation is much easier. Companies react to confidence (whatever the source) the same way individuals do.
Also it sounds like Uber needs to fire it's CTO.
Take chat, hipcat is like 5-10 bucks per month per user. Uber has about 20k FTEs, 40k contractors all use chat. That's 3 to 600k per month just on chat.
Don't forget scale is expensive.
I can't imagine how much that many slack licenses would cost.
Except the people who were laid off, if they have been there a while, got much deep knowledge than if they just glued together a bunch of AWS stuff.
Gluing together AWS stuff can get you a good job, but it's not the same as solving these sort of problems.
I'd rather spend 3 years working on this and get laid off than the way I spent my last 3. Guarantee I'd be in a much better spot in the job market too.
Instead I'm trying to figure out how the fuck I get my career back on track because I'm headed towards the sort of job I will hate every single day.
But that's just it: the problems were not solved. Personally I would rather devote years of my career building something that actually works by standing on the shoulders of giants, than going full NIH and wasting those years on a clunky POS that will be mothballed before the decade is out.
But hey, I chose science.
I remember doing that too, once upon a time. I invented a templating system based on Lisp for building web apps, back in the days when there were servlets, but no Java Server Pages (or Handlebars, or jsx, or anything we have today).
The templating system was wonderful, and it did improve developer productivity for our group, but there is NO WAY WHATSOEVER that it paid back the time I put into it.
The business would have been much better off if we'd simply used what was available and spent the majority of our time solving business problems, not developer productivity problems.
Furthermore, my templating system didn't wind up being that special. You know why? Because I wasn't really solving a business problem with it, so I had no constraints upon my design. Without constraints, such things meander from shiny bauble to shiny bauble according to the engineer's whims and fancies.
When you're solving an actual problem, you have constraints in time, resources, and solution. This forces you to create actual value. That makes what you build actually useful. But when you're building a thing because you want to play with the thing, you have no constraints, and rather than freeing you to create something amazing, this simply sucks you into making something that is worthless for anything except résumé-building.
"Ah, but my résumé," says the narcissist who feels no obligation to create value for their employer. I have bad news, and I am speaking from experience. If you build a bunch of things with way-cool tech that created no value, you become an ignorant person's idea of a smart developer.
Sure, you can get hired by people who don't really understand how to manage engineering, and they will throw money at you, but their ignorance combined with your self-serving ways will result in another disaster. Lather, rinse, repeat.
Good engineering cultures will ask you about your projects, and they will ferret out the fact that your projects did not add value, and did not "move the needle." I am not hypothesizing this, I have gone to interviews and had the interviewers grill me on my vanity efforts.
You cannot pull the wool over the eyes of anyone who has been through a cycle like this and is determined to never have it happen again. You can put lipstick on a pig, but there's no way to hide the fact that all projects like this wind up becoming ad-hoc, informally-specified, bug-ridden, slow implementations of half of the thing you should have used instead of building your own.
Now all is not doom and gloom. For everyone who has done this, either deliberately or by being dragged into one of these white elephant projects, there is a way to supercharge your career going forward.
Instead of boasting about your ability to write Lisp-based templating systems, or synchronization protocols based on Operational Transformations, or whatever it is you built, try confessing to it.
Point out the engineering experience, but then be up front that the project was a failure/bad idea. Sure, point out that you learned some great tech. But spend much more time talking about how much you learned about failure modes for projects. Talk about the scars.
Companied worth working for are eager to hire people who know better. Position yourself as someone who has learned these lessons the hard way.
Learning those lessons the hard way still requires doing those projects, which means you learn how the tech works, which means I still feel 100% justified in my previous statements.
I want to work on a project like this that actually has lasting value to the company. Actually does something novel enough that it's worth doing.
How do I get there? Is the only way to win the lottery and get placed there as a new grad/junior engineer?
Cause that seems like what you're saying.
I know it's very hard to get jobs in established good companies, but there are also quality startups, founded by people who have these kinds of scars already and won't go down the path of building things nobody needs.
Here's a specific tip:
When interviewing, there's a point where you get to ask them questions. Great! Ask about projects like these. Is the company working on any of its own tech that could otherwise be rented from Amazon or is available as open-source? Why? What is the expected benefit? If not, how do they manage development to stay focused on projects that deliver value?
By asking these questions you will identify engineering groups that are strong or weak, and you will also send them a strong signal that you care about success. Good companies will pick up on that signal and be more inclined to hire you.
JM2C. My advice plus $3.00 will get you a latte at Starbucks.
----
* This is not a dichotomy, of course.
Most of the things we should use instead of building our own are either currently someone's shiny, golden side project with no business case, or else started that way.
In my experience this is one of the most toxic diseases an engineering org can catch. It often comes from smart, technically capable engineers who are more concerned with impressing their peers than growing the company. They come up with a BS reason to re-implement some rock solid 3rd party tech, spend ages doing it, only to leave the company as it starts to comes crumbling down. At that point other engineers have to put in massive effort to rip out their buggy timesink, and replace it with the open source library/database/protocol/whatever that was the obvious choice from the start.
If people like this exist in your org, you either need to convince them of the error of their ways, or let them go. The damage they cause when left unchecked is just too great.
Well, if the company's business is connecting drivers with riders, there isn't much engineering involved in growing the company. Pricing, handling investors, PR/advertising and working with municipal governments does affect your growth. The exact details on how you store the destination's GPS coordinates in the database don't really matter. A good product manager who understands usability and can communicate their ideas to the management can add some value, but having a 20% lower price than the competitors do will make 90% of the users ignore any usability hiccups.
Hiring thousands of engineers to maintain a process that only needs a handful is perfectly reasonable, since the investors don't understand tech and generally like the head count. So this is critical for raising VC capital needed to subsidize the rides, that in turns gives great growth numbers.
A typical developer doesn't understand these mechanics. And even if they did, the only alternative to delivering useless unneeded results would be to resign, except good luck finding a competitively paying non-bullshit job in the current market.
All have had the same experience re: there being more feature work and tech debt elimination than they have the resources to complete.
So, whenever a technical team identified a need for X, there was a lot of time putting into understanding the problem, understanding the company's real needs, and figuring out the simplest thing that would get the job done. If there was an off-the-shelf product that did one thing and did it well (e.g., protocol buffers for serializing datagrams), that would get chosen. If everything out there was a confusing pile of feature creep that was trying to be all things to all people (e.g., configuration management), an in-house product would get launched.
TBH, the tech stack was a joy to work on. Especially that home-grown configuration management system. So simple, so effective.
I've never worked for Uber or anything, but I've attended a few lectures and conference sessions where Uber engineers presented things they worked on, and my sense was that that wasn't at all what was happening there. Most the projects I saw seemed to be motivated by resume-driven development. Which is absolutely the way to go if you want to end up with solutions that are complicated, hard to manage, and half-implemented.
For example, automated package management was banned at the company level. Instead there was a central repository of approved packages that everyone used. That's actually another great example of figuring out the simplest solution that would work: It ensured internal consistency in a way that more-or-less eliminated a problem that is more commonly solved with baroque solutions like containerization. It gave tech ops a really, really easy place to go look when they needed to do security audits on our dependencies or for pushing security updates. And it made it easy for legal to keep an eye on compliance. Stuff like that. Honestly, I thought it worked really, really, amazingly well. And I'd have never thought of it, because it flies in the face of 20 years' worth of the entire software field pulling things in in a completely different, much fancier and more complicated and bell-y and whistle-y, direction.
Also, most of what was going on was so simple you wouldn't even be able to turn it into an open source project. Like, I know that the official wisdom is that you're not supposed to route your event streaming system through a database, but wow, assuming you're willing to shell out for a commercial RDBMS, it's amazing how far you can scale an event streaming mechanism that routes everything through database tables. And don't even get me started on how easy it was to support in production. Sometimes I have to remind myself, open source is only free when your programmers aren't drawing a paycheck.
Do you mean they had a private dpkg/yum repository? That is a well supported and somewhat common setup, and counts as automated package management (your application gets packaged up as a .deb or .rpm with the necessary dependency and installation info).
> It ensured internal consistency in a way that more-or-less eliminated a problem that is more commonly solved with baroque solutions like containerization.
I worked at a company with that kind of build/deployment system. About the time of the start of Docker mania, a corporate merger happened, and the acquiring company, in addition to buying the company I worked for, also bought into the hype, and insisted the build and deployment process go through Docker. Early days of shitty Docker tooling meant a ridiculous amount of time spent by one of my colleagues getting the system working, with the result that build and deploy times went up an order of magnitude, and debugging got a lot harder. Containers are not a substitute for package managers or automated provisioning.
Two things:
All libraries were collected into a single repository in source control that developers would clone onto their machines and use. These were the same libraries that were installed on the production servers. Very consistently, modulo major platform migrations and the fact of running multiple server OSes in production.
Binaries were named and versioned tarballs that were deployed by just unzipping them on the target server. That was all that was necessary, regardless of server OS, because of the abovementioned consistency in what shared libraries were installed on the servers.
With those standards in place, CI, CD, configuration management, all those sorts of things became just wildly simple.
It's a bit like Bazel's golden handcuffs: If you're willing to let go of the basket case of flexibility that C and C++ have traditionally offered, Bazel can deliver you a build system that's just a dream to work with compared to old friends like autotools and CMake.
My company has similar stories. When you look at the amount of requests per second we're actually dealing with, there are plenty of other companies doing something far less "sophisticated" and handling more traffic.
Keep it Simple, Stupid.
Allowing for heterogeneity is a blindspot for a lot of wannabe architects. We have a system where most of the code is dominated by 2 platforms and there is no way to democratically add/test/demo new routes that don't go through one of those two. And every change has to be able to deal with production traffic levels from word one. Similar with deployment management.
In this sort of climate, there is no incentive to ever go back and revisit NIH vs off-the-shelf tools that has now outstripped the functionality of the in-house tool. When the friction is so high it's a non-starter. And when some of those NIH projects started in ignorance (ie, a tool existed and was already mature enough to start using, but the author didn't find it) it represents extra frustration.
At some point your engineers figure out nothing they are working on will present well on their resume, and they either get burnt out or flea.
Shouldn't the fact that they were part of standing up solid reasonable solutions reflect well on their resume, to reasonable hiring managers elsewhere?
Citation needed? AFAIK Uber was using Facebook's Buck and decided to move to Bazel for stacks where Buck is admittedly not good at (e.g. Go). I'm not aware of any org within Uber reinventing Bazel. The web org might be closest to what you're claiming, but even there, it merely wraps over Bazel to bridge over its shortcomings (i.e. package management is typically done "outside" of Bazel, via yarn/npm in web, as seen in the official rules_nodejs and other 3rd party spin-off rulesets)
Yes, that's exactly how it works. It helps you move from manager to senior manager. It helps your boss move from senior manager to director, etc.
Growing your team is management 101 for anyone with a minimum politically astute. So if the message you get "from above" is that the budget is unlimited (which honestly seems to have been the case at Uber) then you dream up any justification to increase your headcount.
This is hilarious, given they Bazel is the public version of Google-internal Blaze
The biggest thing that people who try to roll their own fail on is this right here.
For example: let's say you have a warehouse management system, which tracks inventory movements in a warehouse / fulfillment center that ships, say, toothbrushes to customers. Multiple customers call in with complaints about getting blue toothbrushes when they ordered red toothbrushes. Seeing as how this is clearly not an isolated incident (multiple people are upset about not getting the right toothbrush!), it's up to you to find out where in the whole lifecycle of that toothbrush in your warehouse - all the way from when it was received to when it was shipped - that error might be happening. If your warehouse management system is completely auditable, then you'll be able query the transaction log (given perhaps the affected order and item numbers) to work backwards to investigate where the fault occurred, be it in shipping, picking, replenishment, receiving, the vendor, or what have you.
The "incumbent" off-the-shelf ERP and WMS vendors tend to support this sort of comprehensive audit trail much more thoroughly than those writing new ones (whether as an in-house project or as a startup attempting to disrupt the space), simply because those incumbents have already had their hard-earned lessons on why literally every transaction must be thoroughly logged (whereas the newcomers and DIYers typically have not, or are in the process of learning as they encounter more and more situations where that audit trail is necessary).
Any system, e.g. SAP, that has a proofen track record in that, that is SOX compliant, is a huge benefit in that regard. Also for the company itself, IMHO, as the company is forced to work in compliant ways that just make life a lot easier.
It is scary how many systems cannot be audited this way because the engineers and business owners had no understanding of basic accounting concepts like double entry book keeping.
Systems that are not designed to be easy to financially audit are either going to overbill or underbill customers. It isn’t a matter of if, but when...
That said, a lot of these sorts of systems are notorious for being pretty awful, too. Yes, there are many millions of man-hours poured into them, but that doesn't mean they're good; those man-hours could very well be spent on trying to implement every bell and whistle under the sun while entirely neglecting bug fixes, for example (and - speaking from experience - that is a disturbingly-common example). Also, having a bad UX can (and often is) a fatal problem for such a system (sometimes even in a tragically-literal sense, e.g. with EMR/EHR systems and other medical software); the whole point of business software is to optimize user productivity, so the last thing you want is to make the user's life harder.
So yeah, tradeoffs. If you're gonna fall into the DIY trap, the key would be to not try to replace the entire ERP or entire WMS in one go, but rather to build smaller tools that optimize specific business processes, and make sure those tools stay small and maintainable; any expansion of features should happen through interoperability between tools rather than trying to build a giant bloated tool. If you're thinking "wow, that sounds like the Unix philosophy" and/or "wow, that sounds like microservices", that thought ain't exactly a coincidence.
Skylark (renamed Starlark) is the extension mechanism. It's exactly the same in Blaze and Bazel.
Source: I work on Blaze/Bazel, and I'm the lead for Starlark.
And functionality (and architecture?); I started my career at Google, and am now at a mid-size place that uses Bazel and it's almost identical from an interface perspective
[0] - https://github.com/bazelbuild/bazel/search?q=blaze&unscoped_...
It seems like everyone does that.
and it makes sense cause if given any chance to do it I'd jump on it immediately.
Uber switch to Schemaless from a Postgres cluster, claiming that Postgres "does not scale for Uber". I see irony in that piece of Uber history. Particularly, Uber always claimed that some OS solution "had serious reliability issues". Cassandra in 2014 worked for companies that had orders of magnitude of traffic, yet Uber found it unusable. Similarly, Uber switched from MySQL to Postgres and then from Postgres to MySQL. Each time Uber published an article to claim engineering victory. The truth is that any of the three systems would be perfectly fine for Uber. Oh, didn't someone also claim that Elaticsearch crashed under Uber's load" and JVM-based systems were not usable for Uber's scale? " before Uber ended up adopting Elasticsearch?
What really happened was that some teams in Uber simply didn't want to dive deep and produce objective tech assessment.
The Dunning-Kruger quotient is especially high.
What I've always wondered is, why do people get so excited about working on ideas that they themselves characterize as 'simple'? The level of excitement around these sorts of projects does not match the level of rhetoric. You put the second rate people on the 'simple' stuff. You save some of the meaty bits for yourself.
Which means that in your head, you know that KV stores are actually pretty complicated to get right. You either aren't admitting that to yourself or to the org or both. It just screams cognitive dissonance.
Because if you were brutally honest and up front about it being complicated, you'd select a good one and have your fun elsewhere, say in using or scaling the KV system instead of building it.
I've been in many orgs where engineers like to reinvent the wheel. Just so they can claim ownership over more components, add polish to resumes ("Led project to rewrite X to scale", "built X from scratch") and makes them look busy. When it comes to perf reviews, or when it comes to job cuts, they'd likely survive given they know more / built those components.