Hacker News Reader

Comment by shrumm | Hacker News Reader

Honestly, I think the more important takeaway isn’t the decline of NoSQL. As the article says, it’s the conclusion that’s what’s good for FAANG isn’t necessarily what’s good for your average project. I know this sounds obvious but it’s been a constant source of frustration for me with the hype train.

You could apply the same logic to a whole bunch of tech like Kubernetes, service mesh etc etc and arrive at the same result.

Every tech has a trade off, understanding it is critical. Don’t pick tech spot SOLELY based on FAANG use.

View on HN

From inside FAANG, I have seen many projects over engineered to death. I think the problem is lack of experience combined with the selfish need to make the job more interesting.

Spanner and Dremel/BigQuery are both SQL database in that you interact with them by sending them SQL. Maybe I don't understand the terminology.

Dremel is not a database.

For non-analytics use cases, only in the loosest sense. Much of the "schema" is key-value is binary blobs.

> selfish need to make the job more interesting.

I'll admit I have been tempted to stray down this path in the past. I think for me it was the fact that the company refused to provide any time for training or improvement, leading to the desire to chose tech in projects that would build skills rather than being the most expedient.

Its usually less about more interesting, and more about "show meaningful contributions for performance reviews"

Resume based development.

RDD - Resume-Driven Development

You can find this in lots of places not just FAANGs

I wonder how well "I worked on Google Wave" is perceived on a resume.

Google wave was a very interesting project, I dont remember it being badly engineered. More interesting that another j2ee crud app.

The origin of my comment was that Google Wave was often considered to be "engineering for engineering's sake".

...we say, an then open Teams or Slack because that's where work lives now.

Otherwise known as Promotion Based Architecture. You’ve got to make the problem difficult enough and the solution complex enough to demonstrate you can “operate at the next level.”

The common pattern one sees in such over-engineered mortgage projects is that they usually have a cool-name, even though they are internal use only.

I guess this is because of the hope for it getting open sourced and the dev becoming famous. Also, the less experienced people are, the more they are thinking that they are breaking new grounds.

Yeah... sometimes you get a junior engineer who's drunk the koolaid a bit too much and think they need a bunch of fancy tech for their web dashboarding dealing with 5 req/s.

But I think it's pessimistic to reduce it to people trying to make the job more interesting, and at least with more senior people it's often about reusing technologies internally.

For instance, an application may have a use case that requires very high messaging rates, so they build up a team to operate Kafka clusters. Then later you end up with a bunch of teams using Kafka when far simpler things would do the trick, because it's already available and there's a whole team supporting it, with expertise to debug it if things go wrong. It doesn't look good if you take the system on its own, but in context it's a pretty reasonable decision.

IMHO sometimes this reasoning goes too far (I've seen people suggest we rewrite super relational apps to NoSQL to avoid operating SQL DBs!), but it usually comes from good intentions.

Sometimes engineers have perverse incentives. If you want a FAANG job, it probably helps to have experience with FAANG tech, even if your current company doesn't need it.

It is a very frustrating industry to spend a long time in. A hamster wheel of constantly shifting goal posts for mastery. I wouldn't mind if they were fundamental advances in software development, but it's mostly the same stuff with small advances but massive learning curves of arbitrary, non-transferrable minutiae.

The side effect of this ever shifting tooling set, is almost no one masters anything. All software is varying levels of crap written by newbies, because when the tools change every project you are always a newbie in that tool set.

As someone at a FAANG company right now, your second paragraph neatly identifies a core frustration I have with my job that I haven't been able to articulate before now. Might be time for a change...

> I wouldn't mind if they were fundamental advances in software development, but it's mostly the same stuff with small advances but massive learning curves of arbitrary, non-transferrable minutiae.

I think this is pinpointing the key exploitability in the market. If you have seen enough tech come and go, you can figure out which 98% of this year's idea are the same as the year before, and 40 years ago.

At that point you can start cutting through the bullshit and design things using brand new tech as if you had used it for 20 years already. At that point you're way ahead of the rest of the pack.

(And you can choose not to use the brand new thing, and argue convincingly for why the almost-exactly-the-same 30 year old, more mature and stable, tech is better.)

I mostly agree.

Though you may still develop blind spots and miss a crucial difference in this year's iteration of the same idea.

Firstly software development is not one monolithic job. There are many different jobs within software development.

You can be a systems engineer – understand the computers at a fundamental level – with focus on systems software – building low-level high-performance components/kernels – for storage systems, databases, in-memory systems etc (each of these have a lot of technical domain knowledge specialisation within them). You can spend 10 years in this field and continuously develop/enhance your expertise incrementally – it is quite stable and rewarding experience. A branch of this specialization is distributed systems engineering – where emphasis is on software operating in a distributed cluster over network – which has additional challenges unique to the aspect of being distributed.

You can be an applications engineer – understand application software engineering methodology – with focus on user/business facing application feature development in a scalable software team setting. The key focus here is not so much about low-level computer systems but more about software engineering discipline. It is about the inter/intra-team sport that is developing and continuously evolving a very large application software code base that is alive with changing/evolving user/business features. It is about modeling the functional domain of the user/business/real world. It is about the modern consumer Internet software development techniques – experimentation, live incremental safe feature releases etc. A branch of this specialization is developing application frameworks and tooling (rather than user/business functional features) to make the life of a feature engineer more productive. You can spend 10 years in this field and become an expert at software engineering discipline of churning out quality features on time and on budget.

The experience and learning in the above job families transcend any particular technology stack. The learnings are transferable from one tech-stack/functional-domain to another with relatively minimal effort. This effort is part of the work itself and doesn't turn you into a newbie for having to do it.

p.s: A generalist full stack engineer is usually an applications engineer who is somewhat good at systems engineering and is able to glue together systems to achieve the application features well. These engineers can take a startup from start and through initial growth phase and up to start of hyper growth phase. But there's a scale/performance threshold – where the scale of application deployment grows, performance starts to hurt your users, you need strong systems engineering specialists to fix those deeper systems/distributed-systems problems. Public cloud systems have been continuously raising that threshold since the beginning.

I just learned last week that we're bringing Pega on for some projects. Never even heard of Pega before then lol. Time to sit down with a free account and learn something else that we'll throw away in five years.

5 years is sadly decent run.

Was in a big Javascript project, where any article over 3 months old was useless or wrong. The internals of the app constantly being rewritten to flavor of the month.

This rings so true to me.

I really like mastering things, so I've tried to stick with a technology stack (that I have kind of mastered), but I don't think it's been good for my career.

It's not just engineers either. I've seen new EMs copy and paste ill fitting FAANG development practices into organizations and then bounce off to which ever company championed it not long after.

Major reason why senior engineers/architects go for whatever technology is the hottest in the market (ie used at FAANG) is because it's the safe choice. For every new project, or for a fresh rewrite, one can either go for the incremental improvement over whatever was in use, or for the revolutionary ones that worked for FAANG and open sourced by them. If one were to go with the former, their solution will always be compared against the hypothetical much better one using the latest shiny framework/componenets used by FAANG. It doesn't matter if you made the right choice, because for you to prove you did, you have to again rebuild the application using the FAANG framework and compare the profiling/scalability numbers. Much easier to just go with kubernets microservice service mesh on nosql.

I can't help but wonder if one of the big differences between real engineers and silicon valley software "engineers" is that real engineers don't make decisions this way.

I don't wonder at all: it's (more or less) true.

Other differences that are important:

- Real engineering interviews test skills that are pertinent to the day to day work of the engineers; FAANG (and copycat) interviews tend not to

- Real engineering isn't plagued by fad-following or driven by personalities: it's backed by research and established empirical practice

That said, the profession of "real engineering" has its share of problems: advancement is as political as it is in any other industry and it's got relatively low pay compared to the true value of the output are two of the biggest.

> Don’t pick tech spot SOLELY based on FAANG use.

Agreed, and I'd go even further than that:

FAANG use should be quite far down the list of criteria, since the use cases and engineering culture are so different to most people's.

I’d go even a step further and say FAANG companies, despite what their tech blogs might argue, aren’t at all immune from making very dumb decisions or adopting very silly practices. To follow those same ideas blindly could cripple your company.

In fact - FAANG companies have the resources to make horribly inefficient processes work (in human time or computer time) - I suspect many of the AI powered things are examples of this - and a smaller company will die trying to get it to work.

I don't think ML-powered systems are inefficient[0], so much as they are incremental gains over the pre-existing system that are only justified by huge sales volumes. If you can get a 4% lift in sales by using a complicated ML model with 100k features instead of a handful of basic heuristics, then whether it's worth spending engineering effort on depends on what your sales are.

[0] Inefficient in the sense that they're wasting money which could be easily reclaimed - they're probably going to be improved over time as the state of the art improves, but that relies on the whole field moving forward.

I do some consulting, and one odd consequence of the rise of ML is that surprisingly many companies weren't even doing simple models.

One problem is that technology is sold with stories about wildly successful companies. This storytelling doesn't concern itself with real world use cases.

I can't count the number of times I've heard Docker pitched with "it's what Google uses!", while the truth was that they didn't.

Before that it was MongoDB which was "just like Spanner which was key to Google's success", while in reality it wasn't even similar.

And you should organize in Spotify-like tribes, even though that idea is one person's idea about how they wished things worked, filtered though an eco chamber of conference talks.

As an engineer it is easy to dismiss these ideas when you have enough behind the scenes knowledge. But the point of storytelling isn't to build technology, but to pitch it. It does a good job at that. So use it, but wisely.

I would argue that kubernetes is often overblown for small firms.

But service mesh is getting easier and easier by the day. And brings quite a lot of benefits!

Even within FAANG companies, developers still don't know how to write Sql.

Reply on news.ycombinator.com