Produce working product first, validate the idea, stabilize the business, start generating profit, and then you can start optimizing your costs.
In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial.
> In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivia
This is a misconception when you including roll out as a part of the change too, changing data once its running in production is hard, changing the data structure is even harder and when you talk about making changes in cache which is at the hot path its probably the hardest. Looking at the graph at the end it looks like it took them 4+ months to roll out the changes after optimization.
100% agreement on this. There are a class of optimizations that can happen transparently. Those can happen at any time, and are fine to defer. Not all profiling and scalability improvements fall into this bucket. Some are very expensive to roll out, and ignoring these concerns can cause huge headaches down the line. Not fun to hear, but it’s definitely true. Even with LLMs, this can still be a huge challenge.
Pretty sure they were joking.
I would not understimate what folks in 70s had done.
Changing data structures with that approach just takes its time as you avoid lock-step updates between components. Sure, by definition this type of development and deployment complexity is hard.
However, what I found the hardest is pushing engineers to adopt this evolutionary data structures mindset, and unless you do that right for the full team, someone will sneak in a backwards-incompatible change that blows the entire effort up.
So it is hard, but primarily for different-mindset-needed, and only then for technical complexity.
I think Cloudflare became big only because they were so much more optimized than others that they offered some services for free that others were not offering. If running costs are high, you only burn (VC) cash and then you exit.
I still look after a few VMware estates and a lot of Proxmox ones (that used to run VMware).
Hilariously, VMware is described as "enterprise class", which I can only conclude means MVP and a bit wanky.
Today I repaired a Proxmox HA + Ceph node using boring old normal Linux skills and as it turns out I have 30 years of those. Part way through a remote v8 to 9 upgrade I think I lost comms due to using OpenvSwitch for networking and despite using tmux for the upgrade session. Anyway, the Proxmox ISO was useless for rescue but the classic systemrescuecd worked nicely and I could run dpkg in a chroot.
VMware "used" Linux and never really gave back. I don't miss fixing vCentres and all the other nonsense that "Enterprise" wankery has foisted on me over the years.
Then I actually met some enterprise software, and realised that it means 'expensive', 'bespoke', 'one-off', and usually 'janky'.
Why solve the problem directly when you can abstract everything away into FactoryFactoryImplementationInterfaceFactorys, and have something that is both a memory-hog and completely unassailable to any normal programmer seeking to understand it or make changes?
> "The improvement in speed from Example 2 to Example 2a is only about 12%, and many people would pronounce that insignificant. The conventional wisdom shared by many of today's software engineers calls for ignoring efficiency in the small; but I believe this is simply an overreaction to the abuses they see being practiced by pennywise-and-pound-foolish programmers, who can't debug or maintain their "optimized" programs. In established engineering disciplines a 12% improvement, easily obtained, is never considered marginal"
also:
> "In the late 1960's we witnessed a "software crisis", which many people thought was paradoxical because programming was supposed to be so easy. As a result of the crisis, people are now beginning to renounce every feature of programming that can be considered guilty by virtue of its association with difficulties. Not only go to statements are being questioned; we also hear complaints about floating-point calculations, global variables, semaphores, pointer variables, and even assignment statements. Soon we might be restricted to only a dozen or so programs that are sufficiently simple to be allowable"
In a recent comment I mentioned a youtube interview with Rico Mariani, a performance engineer from Microsoft, and he said that he often got called into projects approaching their deadlines and not meeting their performance goals.
In one anecdote he spent a couple of hours with a team and showed how their design could never meat the goal even with the fastest disks, CPUs, memory, and network. And commented how strange it is if they had spent a day at the start of the project whiteboarding out the design against hardware specs at the start of the project - and avoided months of wasted effort - that would be called "premature optimization".
Oof that hit hard. The last project I worked on suffered from a very similar disease, and it has really taken a toll on me psychologically. To work day in and day out on something that you can prove cannot work is unbelievably demoralizing. From an organizational standpoint, it makes a lot of sense to have an "internal consultant" who can deliver bad news like this. I tried to do it from the "inside" which was a huge mistake--got a negative performance review saying I had a "communication problem" because nobody wants to hear "negativity". I can come off online as kind of an asshole, so this may not seem credible, but I did actually deliver this news in a professional, measured manner. It's just that organizations are allergic to it, and their antibody response kicks in. You need someone who is not affected by the organizational hierarchy (or at least not that branch of the tree) to step in and deliver the bad news without fear of retaliation.
I'm joking...but not entirely. It sounds impressive on a promo packet when you say you've saved 100 TB of RAM / $$$ through whatever technique. But it sounds a lot less impressive when you say if this system grows to this size in x years, I will have saved 100 TB, especially when no one yet knows how large the system will really be in that time or what the cost of RAM will be. I dunno, maybe if you say that x years ago, I made a decision that now is saving us 100 TB, that's kinda impressive, but you're also getting credit for it x years after you did the work. It also doesn't have the implication that it must be inherently complex/hard because some other smart person chose the other way. And there is a bias to care more about recent accomplishments. So I don't really think it'd be valued the same at all.
Also, in general big tech (at least Google) prefers growing the userbase over improving efficiency. Periodically efficiency is rewarded, e.g. when RAM cost suddenly balloons or some big must-have feature has suddenly used up capacity planned for something else. You get rewarded for doing efficiency work on demand, not eagerly.
I once got a $100 peer bonus for finding 100,000 cores that were essentially stranded by an accounting error in another team's migration script.
Better performance = saving money + better user experience.
The problem with optimizations is that you are competing in prioritization with other features. Reducing the baseline cost always has a limit of zero, while the upside from new features is infinite according to your leadership and investors, so it is very hard to argue against.
In general it's challenging to convince a non-tech crowd of the importance of addressing any tech debt unless you can demonstrate a tangible financial impact on the product, such as delayed contracts or customer churn.
I think companies often over-indulge in features nobody wants, needs, or cares about. I quit my previous company because they were forcing us to build something that had single digit weekly active users. It was utterly pointless, driven entirely by some half baked navel gazing harebrained ideas about what a "nontechnical user" might want. But nobody ever asked any real users.
I estimate the company probably blew the greater part of $10M on this bullshit, not counting opportunity cost.
> In general it's challenging to convince a non-tech crowd of the importance of addressing any tech debt unless you can demonstrate a tangible financial impact on the product, such as delayed contracts or customer churn.
People like that are problematic not just because they don't understand tech debt. They also don't understand products. There are shitloads of people in the industry who market themselves as some kind of mystical gurus, are able to deliver impressive monologues talking over everyone on the zoom call, but contribute nothing else than a sense of urgency and frustration. If you find yourself in their company, better to just leave.
I can easily picture both tech and non-tech individuals acting this way; in the end egocentric people and employees with a herd mentality exist in both groups. I'm old enough to have participated in several major version rewrites aimed at resolving performance and technical problems. More often than I care to admit, these projects failed, often after several delays, and didn't show any real improvement over the existing system, apart from hopeful conversations about the future and tech presentations at local conferences.
The thing I think is absolutely ridiculous is when organizations can't figure out how to get work done which improves the performance (cost, efficiency, speed, etc) of their existing features. "Prove to me it has {financial, customer, etc} impact" is just how managers who care about nothing other than rolling out new features prevent the work from being done. It makes sense, they get promoted when their underlings deliver some splashy new thing. But it's an incredibly stupid way to build software. If the thing was worth building in the first place, surely it's worth making it good, right?
Insufficiently optimized code kills companies in highly competitive markets.
We give perverse incentives to the marketing department to bring us the most customers they can, instead of the most appropriate customers.
Someone told me at my second lead position that they had landed a top-tier customer for our demo-ware and the first words out of my mouth were, "FUCK ME". Not the response they were expecting, but then that guy never did end up understanding me the entire time we worked together. My bosses did though.
When your system is new you're a loss leader for all of your customers. Every dollar they bring in costs you two, possibly more. You have to get to positive MRR before the venture capital runs out, and over a long enough time horizon, you will go for a new round and find out that a recession is about to start and the VC guys getting cagey is the first clue it's coming.
Meanwhile if you spend all of your time and energy on reducing costs instead of increasing revenue, then your competitors catch up with you. Particularly if their funding rounds are half a cycle off from yours - one of you will be the last one to have gotten a cash infusion before the money dried up.
Most recently I worked at a place that didn't understand my calls for sobriety until it was too late. So I got put in charge of a rear-guard action that was too little too late, and our customers all fled to much cheaper competitors who could do 80% of what we could for half the price. I learned some good stuff, but the company got bought by a competitor who scrapped those systems.
<strange high pitched equivocal sounds>
I do a lot of tuning work that others either can't be arsed to do, or more often are scared shitless of touching (learned helplessness). I tend to work on things that have to go right the first time in general, but I firmly believe that I learned that skill doing optimization work early in my career, not the other way around. Through a series of unfortunate events, my first specialization was performance, before I got good enough at anything else for people to listen to me.
Ultimately, the single thing that fucks up performance work, far and above any other concerns or sources, is that the vast majority of developers want to skip from Make it Work to Make it Fast without going through Make it Right. Then they have to use dodgy 'optimization' tricks that embrittle the entire thing, destabilize production and make every new feature feel like walking a minefield.
So what happens is that when someone who actually knows what the fuck they are doing comes along and wants to build a superhighway, they find the route littered with abandoned 'vehicles' that have sunk into the mud up to their door handles and left to cement in place. Clearing the proverbial route becomes a bigger technical challenge than building the road. And so you'll find that your peers have a bunch of ideas about what needs to be done but none of the stomach to do any of it.
Which is why I say "stubbornness" is the most important tool in my performance toolbox, far above cunning.
this assumes you can generate profit before you can get optimized - what if profit generation is only possible with optimized software? A lot of online MMO-style games tend to require such optimizations as they scale into the size required to generate profit.
Or, in the current era of ai, the cost of the capital investment is far exceeding the ability to generate profit off it. The optimization in how the resources gets used will be needed to cut the costs down, and allow increase in the scale of usage for the same hardware. That's where profits would lay.
Of course, in order to achieve any of this, you'd need the runway to survive until such times. A small scale operator won't have this runway, and so die before they can accomplish anything profitable (or get big by begging for investor money to grow large - as we've seen in the past 20 years of tech).
One, most of the things you believe scale linearly scale at nlogn because of physics. Storage is much worse, scaling at n^1.5. What fit in a single 4U case is orders of magnitude slower when it no longer fits in a single data center. Inserting records into a table with a billion rows is much slower than a table with ten thousand. Wipe the surprised Pikachu look off of your face.
Second, every architectural choice needs to be reviewed every time traffic increases by an order of magnitude, and preferably before so that you have time to stew on possible solutions and allow serendipity the time it needs to find you a better answer. If you have made too many architectural decisions, you will find there aren't enough hours in a month to keep up, even if you never sleep. That's the real reason we practice Principle of Least Power. Not security, but job security. Keep the system small enough that you have time to keep the plates spinning while also implementing new features to attract or retain customers.
I mentioned acres of land. You normally don't have multiple acres of land in the suburbs.
> is it possible to buy a reasonably nice home located in a reasonably nice amerikkkan city… for $300k in 2026?
Who wants acreage? We want homes.
What does "city" mean to you? For some, it's 500 people, or 5,000. For some, it's 5 million. Define that first. The US is a big place, and I know people that don't live within 50 miles of another human.
Otherwise:
https://www.zillow.com/homedetails/424-Olive-St-Kansas-City-...
4 bed, 3 bath, 1,580 sq ft, beautiful! $342,500, built in 1900.
https://www.zillow.com/homedetails/3508-N-College-Ave-Kansas...
4 bed, 4 bath, 2,295 sq ft, $365,000, built in 2022.
You'd be surprised how much house and land you can get for that price.
Yeah that sounds cruel and messed up, because it is. I don’t like how expensive real estate is and I wish we could have another 2008 without a bailout. But the point stands. If you hafta pick between eating and paying your mortgage, you made a bad financial decision. Blame whoever you want, I am of the opinion that people still have agency and are responsible for their decisions, as unpopular as that seems to be these days.
So you could afford 200/month more, but they won’t lend it to you if your max mortgage is 300 and not 350.
Acknowledging this isn’t always easy or possible, but just pointing out that this is a self reinforcing problem.
I meant desirable for me to live there, not as an investment. Who wants to buy a home in place they don't want to live?
Tons in the $200s btw, and this is in the city less than 20 minutes from the Nelson-Atkins Museum, WW I museum, the river, farmers market, KC Chiefs / Royals / Sporting KC / KC Current teams, parks, schools, newly rebuilt international airport, etc.
[0] https://www.realtor.com/realestateandhomes-detail/3711-N-67t...
[1] https://www.realtor.com/realestateandhomes-detail/20118-E-Tr...
That said, look around:
https://www.zillow.com/kansas-city-mo/?category=SEMANTIC&sea...
Between $175k and $350k center on KCMO alone there are over 760 results.
Note: your first link https://www.realtor.com/realestateandhomes-detail/3711-N-67t... is 23 minutes from the WW I museum.
Note 2: Your second link is way out in Independence, MO but is still only 29 minutes away from the WW I museum.
First if you us the convention center, you place the housing center off of where most of the housing actually is.
Between $175k and $350k center on KCMO alone there are over 760 results.
I realize multiple acres of land is a bit of a silly requirement, but you're the one who suggested there are tons of houses with "several acres within 20 minute drive to a lot of US city downtown areas". That's why I was surprised.
This is my direct quote.
Let me break this down:
1. That would buy you thousands of square feet
True, clearly seen in the search results.
2. and often several acres
True, you just filtered down to several results currently on the market.
> I realize multiple acres of land is a bit of a silly requirement
Why? I live slightly further than 20min outside of town. There are massive, beautiful homes with tons of land out here, and tons have gigabit fiber from GFiber or several of the other providers in the area.
But that's not a requirement if you don't want it to be! There are clearly tons of homes available in the city with 3-4+ bedrooms and 2+ bathrooms in the $200 and $300ks is the point.
not everybody is so lucky to be able to go in that order? The first part requires upfront capital/investment?
from a business perspective, this might be considered the only way, but it is not. at large volume scale it becomes more, but often large scale is lacking optimisations in the first place.
its not wrong in my eyes, but definitely not the only path to take.
It's only right if you are not the one who pay the bills.
Not if the whole thing is architected poorly but was a requirement of the hour so it became big. Then optimisation becomes an art, but definitely not the ‘easiest part of the process’
Good thing they jumped on that as soon as they were profitable instead of burning cash. Oh wait...
I think a distinction to draw here is that Cloudflare had relatively large capital raises and were almost immediately profitable¹. They had the luxury of throwing away money. Judicious optimisation makes sense for scrappy start-ups, especially when trivial optimisations like these could easily be farmed off to an agent.