Hacker News Reader

Comment by lumost | Hacker News Reader

I suspect the common issue with small scale projects is that it's not atypical for the engineers involved to perform a joint optimization of "what will work well for this project", and "what will work well at my next project/job." Particularly in startups where the turnover/employer stability is poor - this is the optimal action for the engineers involved.

Unless employees expect that their best rewards are from making their current project as simple and effective as possible - it is highly unlikely that the current project will be as simple as it could be.

View on HN

What I've found to be even more common than resume driven development has been people believing that they either have or will have "huge scale". But the problem is that their goal posts are off by a few orders of magnitude and they will never, ever have the sort of scale required for these types of tools.

I had this very same argument today. It was claimed that a once per year data mapping process of unstructured data that we sell via our product - would not scale. The best part is if we somehow had ten of these to do it would still be something that would take less than a year. Currently it takes a single person a few weeks and makes millions of dollars. This is the sort of fiddly work that you can find an Ontologist for and they’re happy to do it for the pay.

I’m unsure what is unattractive about this but I guess anything can be a reason to spend a year playing with LLMs these days.

I’ve had the same problem with compliance work (lightly regulated market) and suddenly the scaling complaints go away when the renewals stop happening.

The problem is when discussing techniques everyone uses the same terms but no one actually defines them.

I think because so many blogs, resources, textbooks etc focus on scale, developers are biased into thinking that they need to build for scale.

Which is wrong a lot of the time! You need to build what is needed. If only 10 people use your project, the design will be entirely different than if 10 million people use it

LLM solves this by meeting devs in the middle: the vibe coded DB schema, coupled with agentically-made application code, makes even 20,000 records a "huge scale".

This is so accurate. I’ve looked in wonder at how someone is maxing out an r6i.32xlarge MySQL DB, when I have ran 4x the workload on an r6i.12xlarge.

Schema design and query design will make or break your app’s ability to scale without skyrocketing the bill, it’s as simple as that.

any blogs/books you'd recommend on schema & query design? it honestly surprises me that these coding-focused models can't look at a schema; look at how data is being queried; reason about the use case for the data; and help prioritize solving for the most likely bottlenecks to scaling the underlying data services.

https://learn.microsoft.com/en-us/sql/relational-databases/s...

This one is a classic for MSSQL, most of it is applicable on postgres.

I like http://www.redbook.io/

At my current job working for a big corporation, a big reason why we use Kafka for non Kafka workloads is that getting alternate stacks approved is annoyingly difficult. Someone already went through the pain of getting Kafka on the company network for their big data use case, and entreprise IT will set it up for us. Using something else for queueing would require way more paperwork.

This is something to catch in hiring and performance evaluation. Hire people who don't build things to pad their own CVs, tell them to stop if you failed, fire them if that failed

Hiring irrational players, or forcing rational people to act outside of their own self-interest is not a winning strategy either.

There is nothing wrong with building stuff, or career development. There is also nothing wrong with experimentation. You certainly would not want to incentivize the opposite behavior of never building anything unless it had 10 guarantors of revenue and technical soundness.

If you need people to focus, then you need them to be incentivized to focus. Do they see growth potential? Are they compensated such that other employers are undesirable? Do they see the risk of failure?

Some people do things for other reasons than maximising their monetary gain at every step; there are all kinds of motivations out there including liking solving problems

There's also a huge spectrum between "pick a job that's good for your career" and "at every step of the way I'll do whatever is best for me, the company and my coworkers be damned"

If you can't see that, just be open with it in the interview process

This is a great way to get only people who basically can't build anything.

The people who use things they don't need to pad their CV haven't actually learnt the interesting parts. The only thing you can count on is that you'll get stuff added to your system to propel the person to their next role

Reply on news.ycombinator.com