Hacker News Reader

> Noone will pay you for invoking existing solutions. They pay you to push the boundary.

Most businesses pay developers to solve problems, not to push boundaries. They typically prefer developers using "boring" tried-and-true approaches.

In my experience it is developers who push for "boundary pushing" solutions because it is more exciting. And of course the vendors selling those solutions.

If it was true that the only way to get "paid" is to push boundaries, 99% of everyone working in IT would be out of a job. Kurt Vonnegut once said that "the problem with the world is that everyone wants to build, and noone wants to do maintenance". I'm reminded of that quote daily working in the software industry -- the people "pushing boundaries" are a tiny minority and if they were the only ones working our modern world would come crashing down in an afternoon.

You can hardly expect people to talk about all the maintenance they did in their performance review

Why?

Work should be balanced between creating new code and maintaining existing code. And managers should value both.

Because the company's internal rubric for performance reviews awards 0 points for this kind of work.

You answered yourself.

Maintenance is seen by business as a cost, innovation is seen as potentially bringing more money. The incentives are all wrong and that translate to engineers (and teams, and companies) chasing features and not fixing bugs.

A rewrite is often the only way of getting enough resources to fix up things.

They should value both but just don’t in my experience. Building a system can get you promoted. Performing maintenance to keep it running generally will not. Doesn’t matter if the maintenance project demonstrates a higher level of proficiency than the original development. Doesn’t matter that it’s extending the life of a system providing known value, as opposed to building something with theoretical value.

Of course they should, but it just doesn't seem to work out like that.

I was basically pushed out of my previous job because I was the bug-fix guy and liked it.

When I was the 'new stuff' guy, I got big raises every year. When I moved to being the bug-fix guy, and other people were the 'new stuff' people, suddenly I stopped getting good raises and barely met inflation. Even my coworkers were heaping praise on someone who I knew was over-complicating things because he was doing new stuff.

As I was leaving, a system that I had rewritten twice was up for another rewrite for more functionality. I would have loved to do it, and I knew I could. They gave it to someone else. I heard, 6 months after I left, that they failed and gave up on the rewrite. I am absolutely certain I could have done it again.

Getting a new job got me a 40% pay raise. It's not that I wasn't still worth a lot more every year. It's that they couldn't see it because it was all maintenance, and not shiny new stuff.

I still prefer bugfixes, and I still rock at them. But I end up doing mostly new stuff and don't say anything because I know how that goes.

It's what we do at least. More than 60% of the work is maintaining old code/systems.

Why? Maintenance can easily be quantified. Fixed x number of issues, implement y number of feature request/changes. Limited downtime to z amount of time by doing following actions...

...why in the world wouldn’t you talk about the ways you made an application better, more resilient, or heck, just able to continue operating??

No, I don’t want my local government official to build a new damn road. I would clap and love them for fixing the one I drive on everyday! Even better if they do it in a way that’s going to last for 100 years instead of 10, and that’s the way you brag about maintenance work.

Yes, put it in terms of churn mitigation, compliance, security, and data integrity and you can absolutely sell maintenance at your performance review

Programming is creating a "Hello World" program. The rest is debugging and maintenance. ;)

> People don't want solutions to yesterday's problems. These are considered trivial and already solved, such as invoking a shell command (which just hides a lot of complexity under the hood). Noone will pay you for invoking existing solutions.

I am not exactly sure what you mean by this, but taken literally; most companies pay well for this and it's also the most common work programmers do all over the world. But maybe I misunderstand your meaning.

Heck, aren't Google devs complaining about "proto-to-proto" work ("protobuff to protobuff", not "prototype to prototype"), where they're basically just CRUDing together pre-existing services? Google pays a lot and has huge scale plus is considered quite innovative (I think they publish a lot of papers each year), yet most of their work is the typical enterprise middleware development job.

> Any random access structured programming language completely breaks down for working with GBs of data, in parallel and on a cluster of machines.

Why would you need a cluster of machines to work on mere gigabytes? You can get single-processor machines with terabytes of ram. Even laptops come with 64GB now.

Yeah, big data these days should be petabytes of data or at most high hundreds of terabytes coupled with very intensive access patterns.

It's almost like the parent comment is just putting keywords together instead of a coherent argument!

> Noone will pay you for invoking existing solutions. They pay you to push the boundary.

I wouldn't put it like that. 90% of DevOps jobs is to invoke the existing solutions quickly. Sure, there will be a little bit of pushing the boundaries here and there was needed. But largely I'd describe those jobs as putting the Lego blocks together every day - and there's lots of money in it.

DevOps Engineer here, former Lead (moved from startup to Big Corp). That's literally my job day in day out. Put the Lego bricks together. Everybody around me tells me I'm doing a good job and I'm sitting here thinking, "but all I did was Google these things and insert some variables in some Helm charts?"

The other part of my job is more difficult, which is encouraging teams to de-silo and automate wherever possible but so many of the employees are threatened by this idea at Big Corp that I'm encountering a LOT of resistance.

I think your first point answers your second point. It's only human nature for people to worry about their jobs.

Same here. It's easy to forget the difference between a master LEGO builder and the average person just piecing together bricks though. Knowing what to build and when/how is a lot of the job too. I try and remind myself that we're also often there to help other people build better things, perhaps analogous to automatically sorting LEGO bricks so other people building can move faster with fewer compromises.

This applies to most high-value knowledge work. Figuring out which problem to solve and how to define success is much more difficult than actually turning the crank to implement whatever solution.

Quickly and cheaply. People who only do development tend to look at every problem as some code they need to write.

We saved multiple customers a lot of time and money by just doing minor load balancer or web server configurations, rather than letting their developers doing a custom application to handle things like proxying or URL rewrites. Similarly we also frequently have to ask customers why they aren't offloading work to their databases and let it do sorting, selection, aggregation or search.

Having just a small amount of knowledge about what your existing tools can actually do for you, and knowing how to "Lego" them together solve a large number of everyday problems.

>People don't want solutions to yesterday's problems. These are considered trivial and already solved, such as invoking a shell command (which just hides a lot of complexity under the hood). Noone will pay you for invoking existing solutions. They pay you to push the boundary.

You'd be surprised. Most software production re-builds and re-invokes existing solutions that just aren't systematized yet. It certainly doesn't push any boundaries...

I think the problem is we keep using the same crappy tools because people are scared to be an early adopter.

Meanwhile, there are things sitting on the shelf that solve these problems in a principled way and make things simpler:

- ML languages

- Nix

- Bazel

- Erlang

- Rust

Some tools that are great and have large adoption:

- Git

- Terraform

> ML languages

What are those?

I suspect he means Ocaml, sml and bucklescript.

agreed. ML [1] is short for meta langauge. Because it is not very C-like syntax by default, it can feel exotic and alien to most of today's unix graybeards, 90s enterprise, aughts startup folks, and even the JS crowd.

see also its 'newer incarnations' Scala, F#, Nemerle...(don't slay me)Rust(ish)

[1] https://en.m.wikipedia.org/wiki/ML_(programming_language)

Oh, I see. Thanks!

I would add E[1] language to that list.

[1]http://erights.org/elang/index.html

I think Nix will take a long while to increase adoption, because it is hard to learn. The language is new, the concept is new, the mental model is new. But you need to master all of those to use the tool effectively.

Same goes for the other items in your list. Git had enough time and force behind it, and I believe the other tools will succeed as well. But it will take time.

For your typical scaling org, I think data layers are often the main issue. Moving from a single postgres/mysql primary to something that isn't that represents the biggest hurdle.

Some companies are "lucky" and have either natural sharding keys for their primary business line, are an easily cacheable product, or can just scale reads on replicas. Others aren't, and that's where things get complicated.

Tbh, That's why for our new projects we've completely ignored relational databases. They're a pain in the ass to manage and scale poorly.

DynamoDB, on the other hand, trivially scales to thousands (and more!) of TPS and doesn't come with footguns. If it works, then it'll continue to work forever.

How do you handle cases where strict synchronization is required?

Banking app is a classic example.

This is funny to me since modern relational databases can get thousands and more TPS in a single node. My dev machine reports 16k TPS on a table with 100M rows with 100 clients.

> pgbench -c 100 -s 100 -T 20 -n -U postgres > number of transactions actually processed: 321524 > latency average = 6.253 ms > tps = 15991.775957

Yep. And 98% of software written today will never need to scale beyond 10k TPS at the database layer. Most software is small. And of the software that does need to go faster than that, most of the time you can get away with read replicas. Or there are obvious sharding keys.

Even when thats not the case, it usually ends up being a minority of the tables and collections that need additional speed.

If you don't believe me, look through the HN hiring thread sometime and notice how few product names you recognise.

Most products will never need to scale like GMail or Facebook.

> people are scared to be an early adopter.

ML was first developed in 1973. Ocaml in 1996 and SML in 1997. Great tools which haven't been popular in 20-40 years probably have something beyond fear of early adoption inhibiting them.

I'd say herd behavior and network effects are the main issues.

Do a quick search on YouTube on "what programming language to learn". You'll find video after video using popularity as a primary or even the primary factor to base the decision on.

Non-technical management tends to do the same, based out of the belief that languages are all pretty much equivalent and a desire to hire easily swappable "resources".

Yaron Minsky joined Jane Street in 2003. The success of Jane Street and the use of OCaml there is well known in finance circles. Finance is full of people who don’t exhibit herd behavior - many firms have a culture of studying fundamental causes and taking contrarian positions. That begs the question - why did OCaml not get widely adopted in finance?

Jet was purchased by Walmart in 2016. Mark Lore from Jet was appointed as CEO of Walmart to turn the company around, particular its technology operations (to compete effectively with Amazon). Jet’s tech was mostly developed in F#. Yet, Lore did not push for its adoption widely within Walmart.

IMO explaining away the failure of ML languages to gain market share over multiple decades as “herd behavior and network effects” is lazy.

> why did OCaml not get widely adopted in finance?

I think it has been pretty successful. The bigger success story in finance is F# though, which takes a bunch of ideas from OCaml.

I think it's tooling. But tooling follows from adoption in most cases.

Agreed. The stuff "sitting on the shelf", as your parent comment said, have problems too (eg tooling). They might solve some problems, but are far from the silver bullets we are looking for.

Building up the tooling is how we advanced over the last decades. It is a very slow process though because one tool has to become stable before tools on top of it can progress instead of adapting to changes.

Large software projects do not have decades. Only few years.

Distributed operating systems are nothing new though. Plan9 for example.

I'd say the problem is way different and it's a people problem. A small percentage of people invest enougu resources to be able to have the big picture and write higher quality solutions. There is a huuge percentage of people with minimal education learning along the way and by learning along the way they reinvent things as a part of their learning process. This also incentovizes rising popularity of low quality but simpler tooling

The number of developers (and developer managers) who want to only complete tasks where someone (else) has defined the inputs and the outputs is extraordinary.

It is my firm belief that the vast majority of value in Software Development is in understanding the problem and the ways that Software can help. Crucially this includes knowing when it can't.

When asked what my job was once I said, "to think and write code, the hard part is doing it in that order."

This is a pet peeve of mine - I've worked with folks who were/are good developers but wanted to constrain their role to only development tasks. No effort to get exposure to build automation/deployment pipelines or understand how Docker/Kubernetes work at a high level, no desire to understand the bigger business domain or how our customers use the product. Pretty much just "give me a fully groomed ticket and leave me alone until I tell you it's done."

On the one hand, I get it - sometimes it actually is nice to have a ticket that you don't have to think too much about the bigger context for, and can just crank out code.

On the other hand, I don't think it's particularly good for your career security if you're limiting your own role to "minimally communicative code monkey" - if that's all you want to do, there's an awful lot of competition out there.

I've made an effort the past couple of years, across multiple jobs now, to get some real exposure to the ops aspects of my roles (most recently, prototyping and then helping implement CI/CD pipelines for Azure Durable Functions) as well as making an effort to understand the business domain, where the gaps in our product offerings are, and what the competitors in that sector look like. It's really helpful in terms of looking for a more efficient way to solve a business problem, not to mention being able to say things like "hey, the market we're in is moving pretty heavily to Kubernetes, so it's really important that we decide how we're going to support that."

I'm not saying you need to be (or can be) an expert in all of those things, but I think having the high level exposure is really important, at least when you get to a senior IC level. A minor bonus is it helps you develop a Spidey sense for when the business is pursuing strategies that don't seem likely to work, giving you a chance to either try to offer feedback/seek clarification on why, or to pull the plug and find a new gig if it seems bad enough.

It's very frustrating and makes it easy for management to outsource your role. Sometimes that's the right thing, but given how little most "ordinary" companies leverage technology and how utterly most non-tech understand what can (and can't) be done it's usually the wrong call.

To put is cynically, everyone knows that yesterday's tools don't work. So it's better to try today's tools, which you don't yet know don't work. This allows one poorly thought out idea to replace another - especially when selling new stuff is profitable.

That said, it's implausible that the main problem is scaling since mainframes scaled considerable data quite a while back.

Agreed completely. Today there's such a large disconnect between how you think about software (APIs, services, systems, data flows, replication, data storage, access patterns, and so on) and how you actually develop software, with plain text files on disk that don't at all resemble our mental model.

I've been working on a new type of framework that is about making how we develop software match this mental model. Where the higher-level primitives we think in terms of are first-class citizens at the language level.

I think only then is it possible to reason about, and scale, software in a much more productive and effective way.

> Today there's such a large disconnect between how you think about software (APIs, services, systems, data flows, replication, data storage, access patterns, and so on) and how you actually develop software, with plain text files on disk

I view this a little differently. People have tried again and again to move the abstraction layer for coding above text files. This was happening back in the 90's (in the guise of so-called "4GLs" [0]) and still today (now rebranded as "no-code" [1]). I myself spent a good deal of effort trying to code "diagramatically" through top down UML (with an amazing product for it's time, Together). So the ambition to shift programming up the abstraction food chain has been tried continuously for 30 years and continued to fail every time. Eventually I changed my view and decided that there were fundamental reasons why higher level abstractions don't work - software is just too complex, the abstractions are too leaky. The details matter, they are critically important. We are in a long arc of figuring out the right abstractions that may take a century or more and in the meantime, we simply have to have the flexibility of text based formats to let us express the complexity we need to manage in the meantime.

[0] https://en.wikipedia.org/wiki/Fourth-generation_programming_...

[1] https://techcrunch.com/2020/10/26/the-no-code-generation-is-...

I don't disagree,but I think there's space for several levels of abstraction between where we are today and no-code tools.

> By tooling I mean programming languages, frameworks, libraries and operating systems. All of whuch have been designed for a single machine operation with random access memory model.

You should learn about Elixir (or anything on Erlang VM).

> People don't want solutions to yesterday's problems. These are considered trivial and already solved, such as invoking a shell command (which just hides a lot of complexity under the hood). Noone will pay you for invoking existing solutions. They pay you to push the boundary.

People are paid to work on products which provide value to the business. There is always a Minimum Viable Product, MVP. Meet that. Exceeding that is the boundary that needs to be pushed, not grabbing the latest, possibly untested, tools off the shelf.

> And random access memory model breaks down completely, because it is just a leaky abstraction. For full performance memory is only accessed linearly. Any random access structured programming language completely breaks down for working with GBs of data, in parallel and on a cluster of machines.

This is why constantly re-engineering (for the purpose of engineering) is not the most useful method. I used to work in FoxPro with databases that were GBs of data. If today those GBs are difficult to handle, which they aren't, then there is a problem with how the stack was put together. GBs are trivial.

Erlang/Elixir would like to have a word.

It is very unfortunate so few are willing to give it a try

I watch Erlang / Elixir. But I really like strong static typing and type safety. So for now, I'm not learning anything beyond basic beam theory, lest I will have to unlearn it later.

Gleam lang at least is definitely something I watch closely and may (may) be the BEAMs breakout moment.