Hacker News Reader

I'm not gonna flame you, but I will note that, as someone who gets paid to use my judgement to decide on the optimal trade-off between quality, time spent on the project, and its future maintainability... I feel like all three will suffer quite a bit with this self imposed "handicap".

This is the main crux of the issue IMO: feature output velocity. With the enforcement of sprint-scale development scope, you really don't have time to iterate on a wide-reaching and supportive base layer of software infrastructure so you reach for tools that will get you what you need within the timeframe demanded by whoever hired you.

As someone who's been fortunate enough to work on and lead these kinds of projects (and watch coworkers work on them) I've come to a near opposite conclusion which is that sprints reveal how bad the ROI on this work tends to be.

The way many of these projects go is someone very smart works with subject matter experts to map out the problem space. The smart person (or people) then go and begin building this set of primitives and integrating the product, adjusting as they go along accruing some warts.

After 6-12 months we have this beautiful tool that improves developer velocity, new features are easy to code; then disaster strikes. It turns out the map of the problem space was wrong! A bunch of things the team believed to be invariants aren't! Suddenly business needs are forcing developers to tear down walls in their beautiful abstraction castle until all that's left is a tangled maze no other developer has a hope of understanding.

Now the project is a millstone around the dev team's neck rather than velocity boost they'd hoped for.

The way these projects more often succeed is some senior engineer pastes together an abstraction layer sends it out into the world. It gets heavily abused for years until finally the team says "We know this sucks, and there's a lot of business value if we make it nice, let's invest a bunch of time in retrofitting this" and fight like hell to make the business case. IMO this tends to lead to better projects and value (though unfortunately many companies make the fight harder than it should be)

So what is the way (are optimal ways) from the beginning?

Ultimately, "It depends" and no answer will satisfy all cases.

My general advice is that software engineering has a lot of well worn patterns for problems, stick to those as much as possible. Their great advantage is that any experienced software engineer will recognize them and onboard quickly, allowing you to focus on those parts of the problem specific to your project/company.

In most cases, whatever common pattern you shoehorn your problem into will suffice for the purposes of the business. It will be ugly, and have warts, but will be generally maintainable and not often touched. Again, if this turns out to be wrong after it's been battle tested and shown value, you can begin to migrate away to something new.

There are exceptions to the above, and many companies don't actually go through the motions of following well-worn patterns even when they think they do e.g. many companies with public APIs make a common set of mistakes we've known how to solve for over a decade and have known they're easy to solve if you deal with them up front.

Fiction and non-fiction writers have been struggling with these issues long before software was a concept. Their solution: "writing is re-writing".

I didn't mean to imply a dichotomy. I don't think an intense planning phase up front solves the problem. But feature-sprints tend to crowd out refactor-sprints because of the demands from above for new features. Downtime working on backend stuff is not perceived by the customer as anything benefiting them. But here, the backend is a codebase no one enjoys working with and the customer is the C-suite expecting X features this month because X or X-1 features were pushed last month.

I am thinking very hard about the CAP theorem right now while working on a billing system for a cloud API right now and it is an absolute joy. No, it won't deploy in version 1, 2, or 3, but it might in version 4, and if it does, it will be glorious.

You can find cool technical problems anywhere as long as you are willing to take the path less traveled.

> You can find cool technical problems anywhere as long as you are willing to take the path less traveled.

After doing that a few times, I'm no longer sure if the reward of tackling cool problems to create more robust, better, faster components is worth the stress of missing deadlines.

Looking back on what value of better work materializes and what is, per YAGNI, usually wasted, I just had a thought: perhaps the right way is to take the easy/dumb way and focus all available time/effort to optimize it for performance - instead of abstraction and extensibility. Because in my experience, nobody ever extends the code the way you envisioned - if they do it at all, they do it by first refactoring it to suit their own idea. And, nobody ever goes back to fix performance. Therefore, making things abstract and extensible is mostly a wasted work - but making things fast pays back for as long as the code is in use.

I am doing this for a particular purpose, though, that no billing system I have seen has. I hate metered API billing, and I don't want to use it. In particular, I don't want a customer running up a $10,000 bill and calling me for a refund. It probably will cost me O($1,000) to give them a refund, between the processing fees and the lost compute, and I will be out that money. Most companies that would ask for a refund also won't pay the bill when it arrives (which is, I think, why AWS is so liberal with refunds).

Instead, I want to do credit-based billing: you buy credits, and when you get to 0 credits, you are cut off (with an auto-refill option for the "metered billing experience," but with strict spending limits). This is, in my opinion, a much better UX than metered billing. From a distributed systems perspective, it's isometric to "metered billing with a hard spending cap" which may be the ultimate version.

The problem with credit-based billing (and why nobody does it) is that if you have a service in multiple datacenters, you have to consistently update a database to make sure that you don't drop below 0 credits, and that is very slow. However, by fiddling with the CAP theorem the way CockroachDB/Spanner do, I think we can do credit-based billing and make it feel like the AP system that metered billing is, and behave like a CP system only when we absolutely need to.

Also, this is basically all enabled by the fact that AWS has a precise time service.

In theory, version 1 of the API will be only in one region, version 2 will have active/passive redundancy, and around version 3 or 4 I want to switch to active/active in several DCs to give low latency and high reliability.

> I hate metered API billing, and I don't want to use it.

> Instead, I want to do credit-based billing: you buy credits, and when you get to 0 credits, you are cut off (with an auto-refill option for the "metered billing experience," but with strict spending limits). This is, in my opinion, a much better UX than metered billing.

How do your customers feel about that? Have you researched if your customers are comfortable spending their money upfront to buy credits they might only use much later (if ever)? For small amounts of money that might not be a big deal, but if we're talking about thousands of dollars that might look different.

Going one step further, challenge yourself to _not_ use stdlib, just raw language constructs.

Need a hashtable? Write one.

You won't come out with a celebrity HN presence, but you may gain enlightened samurai monk status.

Going one step further, challenge yourself not to use raw language constructs, but dive deep into assembly.

Need a conditional? Time to jump around.

I wrote a web server that scales to over 1 million requests per second, and I found out it's much more maintainable, scalable and environmentally friendly for our company.

That sounds like it was a lot of fun but I'm wondering what assembly gave you that was not available in e.g. C? My understanding is that most compilers can out-optimise the average developer, so are you an above-average developer (well I guess you are) or did assembly enable you to do something that was difficult in a higher level language? Or was it more about the challenge (which I'm totally on board with by the way)?

> My understanding is that most compilers can out-optimise the average developer

Do you happen to know where (what source) you got that from? I'm genuinely curious, as to my knowledge, compilers are generally still easily fooled by things that causes them unable to vectorize code or factor out conditional jumps.

To give an example here, see Mike Actons' talk below (from about 43:10 onwards) in which he describes the compiler failures.

https://youtu.be/rX0ItVEVjHc?t=2590

No specific source I could point to really, just that I've been hanging around message boards like this for a long time. I didn't mean to say it's impossible to beat a compiler or they don't have any blind spots, but thanks for the link - sounds interesting :-)

> not to use raw language constructs

Nit: assembly is a language. The phrase you are looking for is probably "structured programming".

Touche

I don't think I you can compare the internet of the 90s to today's internet. The expectations and scale are no where similar.

Challenge yourself to not even use stdlib once in awhile. There are some interesting insights to glean about how much room for improvement we have even at the very bottom. https://youtu.be/BrBb0mqoIAc

I wrote entire systems with Turbo Pascal, and then Delphi out of the box. Many others did the same with Visual Basic 6, or the Microsoft Office 2000 suite with VBA, before the .NET infection took hold and Microsoft lost it's mind.

All before Google, Youtube and Stack Exchange