Comment by NumberSix | Hacker News Reader

Comment by NumberSix | Hacker News Reader

Google is highly non-representative of businesses and technology businesses in particular. It has a near monopoly on the search business and has enormous amounts of money from a single source -- advertising.

Google Fiscal Year 2015

Revenues: $74.54 Billion (source: Google) Advertising Revenue: $67.39 Billion (source: https://www.statista.com/statistics/266249/advertising-reven...) Profits: $23.4 Billion

Market Capitalization: $570 Billion

Google Not a Startup/Has Not Been a Startup for Many Years

Founded: September 4, 1998 (19 years ago) IPO: August 19, 2004 (13 years ago)

Number of Employees (2015): 57,100 Revenues per Employee: $1.3 Million Profits per Employee: $409,000

The issue is that Google has so much money and is so successful that it can do all sorts of things that are extremely inefficient, even very harmful and still do fine, unlike smaller companies or startups in particular.

For example the arxiv article states:

2.11. Frequent rewrites

Most software at Google gets rewritten every few years.

This may seem incredibly costly. Indeed, it does consume a large fraction of Google’s resources.

Google has the money to do this. As the article argues, it may work for Google. On the other hand, Google has so much money and such a dominant market position, it probably can keep succeeding even if continual code rewriting is actively harmful to Google.

In orthodox software engineering theory, competent software engineers, let alone the best of the best that Google claims to hire, should write modular highly reusable code that does not need to be rewritten.

Many rewrites and "refactorings" are justified by claiming the "legacy" code is "bad code" that is not reusable and must be written by "real software engineers" to be reusable/maintainable/scalable etc. One rewrite ought then to be enough.

Even highly successful businesses, for example $1 Billion dollars in revenues with $50 million in profits and 5000 employees (revenues per employee of $200,000), have nowhere near these vast resources --- either in total dollars or per employee or product unit shipped. Blindly copying highly expensive software development processes from Google or other super-unicorn companies like Apple or Facebook is likely a prescription for failure.

There's a meta-lesson about the structure of the software industry, though:

Different market niches in software have wildly different sizes & profitabilities. It's worth doing everything you can to get into a market that is both huge and wildly profitable, and once you're there, you should do everything you can to defend that niche from new entrants.

Pretty much everything in Google's strategy - the constant rewriting, the expansion into adjacent markets like web browsers and mobile phones and ISPs, the highly-paid employees, the 20% time - follows from this. Also, most of the counter-intuitive advice in the startup world - the obsessive focus on growth, the companies that shut down because they aren't growing fast enough, the existence of venture capital and willingness to accept significant losses of control in exchange for capital infusions, the use of disposable coding practices and technical debt - also follows from this principle. A lot of things about the software industry that seem stupid, irrational, or short-sighted become perfectly rational when you understand the market structure of technology.

I agree. Many companies, including mine, run critical stuff on applications that are over a decade old that do just fine. Worst thing you can say about them is that some weren't designed well enough for easy extensions or integration with newer apps. They do their job, though. The backbone of ours is mainframe code running on mainframes and AS/400's with simple, terminal interfaces. Terminal stuff is ultra-fast & reliable but sometimes ugly. Some have GUI apps that basically hide the terminal details they interact with to be a bit easier to use. Those terminal apps, probably 20+ years old, still work and get periodically updated. New people learn them easily, too, since interface was well-designed for the time. Can't goof off on the thin clients either as there's no web browser or native apps. ;)

I've seen Google do some rewrites that make sense when one has the money for them. The shift from eventual to stronger consistency in their databases via F1 RDBMS was impressive. Worth some rewrites across the apps to use it to knock out major problems that could affect them once and for all. After developing Go, they also might want to rewrite performance-critical apps in, say, Python to it. There's definitely benefits on such rewrites. A lot of the other stuff I'm betting they could've done more long-term esp if using and extending FOSS solutions.

>Blindly copying highly expensive software development processes from Google or other super-unicorn companies like Apple or Facebook is likely a prescription for failure.

Yes, this can't be stressed enough. Out here with the little people like non-Google-employed me, there's almost a desperation to copy the things that Google and Facebook do, and it's often justified with "Well, Google, Facebook, or Netflix use it!"

A huge amount of tech fads over the last decade have been the direct result of Google's publications and public discussions. Google might need BigTable, but that doesn't mean we should all bury our SQL RDBMS and use a crappy open-source knockoff "based on the BigTable paper". More than likely, Google would've been happy to keep a SQL RDBMS.

Google has the engineering staff, including two extremely technical founder-CxOs with PhDs on which Google's initial technology was based, and the hundreds of millions of dollars necessary to do things The Google Way. They can rewrite everything that's slightly inconvenient and do a lot of cutting-edge theoretical work. They have the star power and the raw cash to hire the people who invented the C language back in the 70s and ask them to make a next-gen C on their behalf.

Google has the technical and financial resources to back this type of stuff up, provide redundancy, put out fires, and ensure a robust deployment that meets their needs. They keep the authors of these systems on-staff so they can implement any necessary changes ASAP. In many cases, Google is operating at a truly unprecedented scale and existing technologies don't work well, so that theoretical cutting-edge is necessary for them.

None of those things are going to be true for any normal company, even other large public ones. Google's solutions are not necessarily going to meet your needs. Their solutions are not necessarily even good at meeting their own needs. Stop copying them!

I'm so. sick. of sitting through meetings that have barely-passable devs making silly conjectures about something they heard Google/Facebook are doing or that they read about on a random hyper-inflated unicorn's engineering blog.

You want your stack? OK, here it is: a reasonably flexible, somewhat established server-side language (usually Java, .NET, Python, Ruby, or PHP; special applications may need a language that better targets their specific niche, like Erlang), a caching layer (Redis or Memcached) that's actually used as cache, NOT a datastore, a mainstream SQL database with semi-reasonable schemas and good performance tuning, and a few instances of each running behind a load balancer. That's it, all done. No graph databases, no MongoDB, no quadruple-multi-layered-massively-scalable-Cassandra-or-Riak-clusters, no "blockchain integration", no super-distributed AMQP exchanges, ABSOLUTELY NO Kubernetes or Docker (use Ansible), none of that stuff.

Just sit down and get something done instead of wasting millions of your employers' dollars trying to make yourself feel important. If you do need an AMQP exchage, bolt it on after the fact to address the specific issue at hand, once you know why you need it, and make sure you know what you're doing before you put it in production (it's apparently difficult to grasp that AMQP servers are NOT data stores and that you can't just write directly to them and wait for some worker to pick it up if you care about data integrity).

Don't start a project with the primary intention of making a convoluted mess; let it get that way on its own. ;)

Well said, this is the comment of the year.

While I agree with the spirit of your post, I think you are taking the argument too far in places. For instance, we are a small team of devs and Docker solved a real problem for us. Our deployment was once on bare OS and it was a nightmare. Different distributions, different library versions (with their own sets of bugs)... Unfortunately we couldn't demand a specific environment from our customers and at the same time it drained our resources trying to accommodate all installation variants. When Docker came out we tried it out, saw that it solves all these issues beautifully and never looked back. But we use it only because it insulates our installation from Linux (apart from the kernel that is), not because it is hip.

More general advice is: take G, A, FB, NF... papers (and HN posts, while we are at it ;) ) with a grain of salt, test before use and make sure it solves more problems than it creates - for you.

Yeah, I don't mean it to be an all-encompassing prohibition. It's just important that people start with the basics and complicate their software with esoteric solutions only as a matter of necessity. This also ensures that the engineers understand the functionality that the esoteric solution needs to provide instead of making grossly incorrect assumptions and misuses that lead to major disasters (e.g., treating AMQP servers as a data store; a client lost data last week due to this silly assumption).

I think there can be many rational reasons for 're-writes' that preclude any assumptions about 'bad code'.

New infrastructure requirements, languages, performance considerations, markets, customers etc. etc. - they all change.

An expanding and adaptive business will likely have to refactor code that was 100% perfect in 'context ABC' because they have some new 'context XYZ'.

There are so many examples to think of.

When they went from one datacenter to many, whenever they replace the hardware architecture in their datacenters or change newtworking configurations. When they decided to integrate G+ into everything. When they decide on using a new common UI framework.

The 'roll on' effects may necessitate refactoring of other code.

And this is not including more opportunistic things in a given area.

Maybe they started doing some stuff in 'Go' and wanted to refactor other modules into 'Go' for greater maintainability?

And I don't think anyone will be 'blindly copying' Google's processes.

Reply on news.ycombinator.com