Hacker News Reader

I used to work with an old Sun Microsystems dude, he was an executive at the company I was at. We used to have these meetings every week and he ended up attending one. We had been trying to come to a conclusion for weeks on a specific piece of tech. He stopped the meeting dead in its tracks and said we're going to make a decision right now, if it's wrong, we'll learn from it, if it's right, awesome. Not making this decision is more costly than making the wrong decision.

I just remember thinking, finally, someone with some authority is getting this ball moving.

Reminds me of a story that happened at my company.

We needed and had purchased a rather expensive database software license, however, we didn't have the hardware yet to run that database. The guys doing hardware spent MONTHS debating on which $10k piece of hardware they'd pick to run the DB. The DB license cost? Something like $0.5 mill.

As one engineer said to me "I don't care what hardware you guys get, purchase them all! We are wasting god knows how much money on a license we can't use because we don't have the hardware to install it on!"

I've noticed a lot of people, even ones making $200k+ in salary, are really bad when it comes to making decisions that involve any amount of money.

E.g. I've been in meetings with multiple developers who, if you add up everyone's salary, is well over $1 million/year, debating for way too much time on whether it's worth it to buy a $500/month service to help automate some aspect of devops.

Maybe this wasn't the case for your specific anecdote, but in the scenario I'm describing I got the feeling that a lot of people think about business purchases in the context of their own personal finances rather than in the context of the business's finances. Leading people to be extremely cautious with things like a $10k purchase that would be "expensive" if purchased as an individual and "cheap" if purchased as a company.

In those cases, getting an exec to come in and pull the trigger can help. The exec is used to looking at big picture budgets/strategy, which IC's aren't. (Although I'm sure someone here can come up with another anecdote proving that wrong)

Lol, that reminds me of another fun one.

Every so often my company would provide lunches for the developers. However, they didn't want to spend too much money doing this. So how did they resolve it? They put together a committee of devs to discuss lunch options/etc. Easily 1+million/year of salary in one room debating whether we do Jimmy Johns or McDonalds and how they'd get the food to the office.

For $2000, you can get some pretty nice catering for 100 people. But like you said, people just seem bad at thinking of that sort of big picture.

Providing daily lunches to developers is an insane ROI and I don't know why it's not standard practice. It's pennies compared to their salary, and they are happier, spend more time thinking about work instead of what/where to get lunch, spend more time eating together and conversing, and probably eat healthier food which mitigates the post-lunch coma.

+1 to this, I had one job where lunch was provided and it very much brought people together, particularly across team boundaries / job levels, and often "tricked" you into having pseudo meetings over lunch.

I find that still happens organically in smaller companies without, but in larger companies things trend towards more clique like behaviour without it (caveat small sample size)

Used to be something that companies just generally recognized as a good thing. Look at workplaces built in the 60 to 80s and basically all of them had cafeterias as part of the building plan. [1]

[1] https://www.nytimes.com/1985/11/24/nyregion/company-cafeteri...

They were generally paid cafeterias though.

When I worked at a merchant bank in London in the 90s there was a very good cafeteria with great food that was not free, but it was so subsidised that it might as well be.

Subsidized. The trick was striking a balance in making the food cheap enough that the employees would eat it but not so cheap that the company is footing the entire bill.

In part since it's a taxable benefit to the employee, absent extenuating circumstances. Not all employees want to "pay" for "free" food.

https://taxnews.ey.com/news/2019-0493-employer-must-substant...

Does this really get reported to the IRS in practice? Most companies provide free coffee and I've never seen that reported to the IRS either.

Snacks have always been okay...from that link--

"Snacks

The TAM separately considers whether the value of snacks provided to employees is excludable from income under Section 119. Rather than revisit the business reasons previously analyzed, the TAM relies on Tougher v. Commissioner, 441 F.2d 1148 (9th Cir. 1971), to determine that the snacks are not meals. Accordingly, a snack cannot be a meal furnished for the convenience of the employer. Nevertheless, the TAM does conclude that the value of the snacks is excludable from employee income as de minimis fringe benefits under Section 132(e)(1)."

Same thing with hardware in every company I've worked at. Really, "Sr. Staff Engineer Alice" makes $300k/yr, but only gets a budget of $2k for a laptop that's meant to last four years and is their primary tool? How does this make sense?

I see devs that are paid a fortune and they use these silly 20 or 24 inch monitors and I don't understand it. A large 4K monitor is a very cheap and really increases productivity. Give them 3 ffs.

Meanwhile, there is an accountant somewhere that thinks he's a genius for keeping the hardware budget in check.

Another good one: my company periodically forces all software engineers to wipe their workstations / laptops and re-install everything from scratch. I have one such wipe coming up, with no way to avoid it, will probably spend 3-4 days just setting up all software again and do no work.

Yea, people generally don't think about these things. People were genuinely surprised to find out that 70% of our expenses as a software business were... salaries. The next largest category? Rent. Everything else was effectively a rounding error.

Internalising what's a reasonable business expense compared to personal expenses is a skill I feel I've only fairly recently developed.

It's super important to consider the human cost and opportunity cost of each decision, and it's scaling characteristics.

Eg: spending $20+k on a ci system like circleci, GitHub actions, whatever might feel like a big purchase, but if you consider that split by the number of developers using it, their salaries, and that in general it scales with your head count rather than user count suddenly it's pretty attractive.

On the other hand some other seemingly small (unit) cost that applies per user of your product might be worth optimizing/eliminating as that can have a big impact on your margins - but even then you need to balance it against opportunity cost. We could increase margins by x% with y investment, or increase revenue by z doing something else where the delta of revenue outweighs the better margins.

Basically it's a big juggling act involving numbers that we're not used to dealing with in our personal finance and you need to ground your thinking in terms scaling characteristics and the companies overall revenue/burn to appreciate what actually moves the needle.

At least they didn't throw it onto some wholly under-specced machine, then yell at OPs because the response speed was measured in days.

When I was a hardware product manager, there were a ton of decisions people asked for and a heck of a lot of them basically didn't matter. If we needed to course correct, we'd course correct. The main thing was not sitting around twiddling our thumbs for weeks or longer.

I mean isn't all of that obvious?

A crap boss is one that doesn't make choices, a good boss is one that does, a great boss is one that makes sure that the best of the possible choices is made giving the data at hand.

Having careers and heads depending on making the wrong choice just pushed to paralysis.

The Google Fiber project was always meant to push the carriers into competition. Google knew that if they didn't launch Google Fiber, none of their other ventures or the internet as a whole, could be as successful. Google paid big money for YouTube and the plan was always to turn it into the service it is today. At the time, there were also worries whether the carriers would restrict services (aka net neutrality) or if they would charge by GB. Launching Google Fiber made it such that the carriers had to start competing and upgrade their infrastructure.

If it wasn't for Google Fiber, I'm certain that we'd be stuck with 20mbps speeds, the cable/DSL monopoly, and we wouldn't have the likes of the OTT services and the choices that we have today. Or at least it would have been delayed by quite a bit.

I worked for a company that was an equipment vendor for Google Fiber and other service providers.

Plenty of countries have better (faster and/or cheaper) broadband options than most of the US, without having any Google involvement. Competition (or government enforced requirements and price caps) are what's needed, Google Fiber had a bit more of an incentive than most for aiming to undercut their competitors but ultimately I think you're overstating their importance.

Competition would be nice, but just the appearance of credible competition was enough to induce the incumbents to do better.

Google Fiber deployed to the Kansas Cities, making themselves credible competition. Then, they announced 20 cities they would deploy to. Suddenly, incumbents in 20 cities had deployment plans and deployed before Google Fiber got anywhere, and then Google Fiber decided not to do any new deployments.

Would the incumbents have deployed without Google Fiber's credible competitive announcements? Maybe? We'd need inside information to know for sure. It sure doesn't feel like they would have though.

> Would the incumbents have deployed without Google Fiber's credible competitive announcements?

Of COURSE they wouldn't!

If google fiber hadn't happened, all providers would have continued sitting on their collective asses, soaking as much money as possible, doing the least possible legally permissible work, nickle-and-diming customers as much as possible.

A commoditize your complements strategy that worked!

Usually those countries have some combination of lower labor costs, higher density (you can run fiber and then hang 5x as many subscribers off it) and a more lax regulatory landscape (try getting permits to dig in a US city).

> Plenty of countries have better (faster and/or cheaper) broadband options than most of the US, without having any Google involvement.

Those countries have governments willing to regulate for the benefit of the consumer, or else to provide the service directly[1]. That there are better ways to do something doesn't mean it's not valuable to have done.

[1] Almost nowhere, in any market, had competing gigabit landlines in residential areas over the timeframe discussed. "Competition" is absolutely not the solution here.

Most countries have policies that expressly prohibit competition, or make it unnecessarily expensive.

Suppose the government owned the utility poles or trenches along the roads, paid for them in the same way as they pay for the roads, and access to use them was provided to all comers for free. All you have to do is fill out some basic paperwork and follow some basic rules to make sure you're not cutting someone else's lines etc.

People would install it. You -- an individual -- could go out and put fiber in the trench on your street, wire up the whole street, pool everybody's monthly fee and use it to pay for transit.

The reason people don't do this is that it's illegal, or to do it without it being illegal would require millions of dollars in legal fees and compliance costs and pole access charges.

You don't necessarily need competition either. Switzerland's state owned telecoms provider provides 25gbit symmetrical fibre to practically all homes in all cities and it is very affordable.

That falls under my "or government..." point, no?

That seems a very US-centric way of seeing the internet evolution.

The rest of the world moved to higher speeds and didn't count Gabs (except on mobile) decades ago and I mean decades.

In 2004 in Italy I had a 20 Mbit/s fiber connection, I had 100 Mbit few years later. I still remember pinging 4, literally 4 ms, on Counter Strike 1.6.

And Fiber was started way later in 2010. So I don't see any impact by Google fiber on internet as a whole, maybe it pushed US carriers to not do worse (internet in US is not really that amazing in terms of speeds and latency).

One thing that I noticed is that while speeds increased in the decades since then, latency became worse. Even with the fastest connection I can use I rarely if ever ping below 30 Ms on the very same Counter strike 1.6 or newer versions.

> If it wasn't for Google Fiber, I'm certain that we'd be stuck with 20mbps speeds

Are you trying to say that Google Fiber influenced the behaviour of incumbent telcos in different regions? If same region, sure, but the size of area served by Google Fiber is/was tiny.

And timing-wise, didn't Verizon FiOS launch like 5 years prior?

> I see the same problems at Boeing: once engineering-driven people companies get taken over by finance leeches. This usually follows an actual or virtual monopoly, just as Steve Jobs described [1].

There is a nice mental framework for viewing such things. It has a bit of a religious origin, but it effectively explains and describes what you're seeing (I'm viewing it through an atheistic lens). I mean, the egregore.

This is the natural life cycle of an egregore! Which is explained by having two groups, those that serve the purpose the egregore was created for (engineers, people that provide value), and those that serve the egregore itself (financials, people that extract value). Both these groups need to exist for a healthy entity to exist. But the balance (seems to) always tip - the egregore eventually chooses the group that serves the egregore to lead - when that happens, the original vision is often lost, and the company looses customer trust by altering the relation the customer has with the egregore (how much value the customer extracts from the egregore vs how much value the egregore extracts from the customer).

https://en.wikipedia.org/wiki/Egregore#:~:text=Egregore%20(a....

This pattern comes up, an possible indication of this flip: when the original owners of a company are pushed out, or leave.

from this recent thread, which is rather relevant: https://news.ycombinator.com/item?id=39491863

> leadership couldn't decide if the future of Internet delivery was wired or wireless.

Weird way to think of this problem (IMO). I'd think there would always be a mixed wired and wireless world.

Even if customers don't end up using wired connections to their homes, you'd still need wired connection to the antennas servicing a home, neighborhood, apartment building. That's where a lot of Telcos today are making their money. Not to the customer, but to Tmobile or At&t as the put in a fiber line directly to the antenna towers.

And even if google wanted to be the end to end ISP for someone, they'd benefit from a vast fiber network even if they later decided it wireless was the best, because they already have the fiber wherever they'd need their wireless antenna.

The last mile is expensive. Even hooking up the customer to the line running outside their house is expensive. I've seen different customers estimate this at anywhere between $2000 and $5000 per premises. This assumes ~40% customer take-up rate so with more competitors, the cost to each goes up. It's one reason why overbuilds make no sense and municipal broadband is the best model for last mile Internet delivery.

Wireless bandwidth keeps going up. Wireless is already >1Gbps inside a building. What if instead of spending $5000 per house, you could use tightbeam wireless or highly cellular network with >1Gbps bandwidth? You may have spent billions on a network that it would take decades to amortize and have it be made worthless by wireless last mile delivery.

> Wireless bandwidth keeps going up. Wireless is already >1Gbps inside a building. What if instead of spending $5000 per house, you could use tightbeam wireless or highly cellular network with >1Gbps bandwidth? You may have spent billions on a network that it would take decades to amortize and have it be made worthless by wireless last mile delivery.

I'd presume you'd not cut the existing wired customers over to wireless. So it's not like the $5000 spent is lost, it's just that you can do new customers for cheaper (if you expect a lot of growth in an area).

Overbuilds is a weird one. They can make sense if it's a brand new community as you can get a BUNCH of homes done for cheap and can pretty much immediately turn on internet when someone moves in.

If the wireless last mile option costs $500/household then you'll get destroyed by a competitor who can come in and offer the same or better service at substantially lower cost. So yes, it does matter.

I'm not sure what you mean by "overbuild" here. I mean it in the sense that both AT&T Fiber and Verizon provide services to the same homes. US policy notionally tries to encourage this (because, you know, markets solve everything) but if you have 100K homes and 40% of them get service regardless, having 2 competitors means each ISP has to recoup their costs from half as many homes.

I'm talking about wiring brand new builds for an ISP. Getting the coax, fiber, telephone line right to the home before the lawn is sodded and drywall is installed is a really fast/cheap job. It costs basically nothing to do because there aren't a whole bunch of easements and property rights problems to navigate.

Generally, a developer already owns the rights to everything so it's just working with them to get everything done. And they like it because who doesn't want an internet ready community to sell?

You end up talking $100 per unit vs the $2000 or $5000. Which is a great discount for the ISP.

Moreover, builders often pay the cost to hook homes up to fibre during construction. Locally (in eastern Ontario, Canada) I have seen the incumbent quote and get $500k to fibre up new rural subdivisions.

> If the wireless last mile option costs $500/household then you'll get destroyed by a competitor who can come in and offer the same or better service at substantially lower cost.

That's not even necessarily true. The $2000-$5000 is a one time cost, and then that piece of fiber could last for 50 years. Having to amortize $40-$100/year against a service that costs around that much a month isn't fatal. Meanwhile they're offering service for whatever maximum speed so you set your price for that $1 below theirs and then charge $10-$20/month more for double that speed which they can't offer at all. Half the customers take you up on the higher speeds and you make back your costs, the other half take the $1/month discount and your competitor is the one who gets destroyed.

> Was it EUV? Was in X-ray lithography? Something else? Intel seemed unable to commit.

Per the book Chip War, Intel put a lot of money into EUV (going back to the late 1990s):

* https://en.wikipedia.org/wiki/Chip_War:_The_Fight_for_the_Wo...

Per the book, and other sources:

> Intel seemed primed to dominate the chip industry as it transitioned into the era of Extreme Ultraviolet Lithography (EUV). The company had played a pivotal role in the development of EUV technology, with Andy Grove’s early investment of $200 million in the 1990s being a crucial factor.

* https://techovedas.com/intel-lost-decade-5-reasons-why-chip-...

I know nothing, but it felt like intel paid the price of being the first. They picked something hard and pricey.. and it didn't pan out in time, allowing other competitors to catch up and adapt to markets (mobile) nicely.

> I know nothing, but it felt like intel paid the price of being the first.

As the book goes into, there were other things in question: since TSMC only did fab, and did not design, they had more customers/opportunities to iterate the process and get good at it (more focus).

There was internal-to-Intel stuff that led to lead loss as well.

I'm only partially through the book currently, and there's a lot of chip history being described (going back to the 1950s), so I'm not going to retain all of it in a single pass.

> I believed in the project and it did a lot of good work but here, ultimately, was the problem: leadership couldn't decide if the future of Internet delivery was wired or wireless. If it was wireless then an investment of billions of dollars might be made valueless. If it was wired and the company pursued wireless, then this would also lose.

This one is particularly amusing because the difference is primarily a business distinction and not a technical one.

Here's how your tablet gets internet via fiber: There is a strand of fiber that comes near your house and then you attach an 802.11 wireless access point to it. Every few years the latter has to be replaced as new standards are created.

Here's how your tablet gets internet via 5G: There is a strand of fiber that comes near your house and then the telco attaches a cellular wireless access point to it. Every few years the latter has to be replaced as new standards are created.

They should have just built the fiber network and put cell sites on some of the poles. Then you sell fiber to anybody who buys it and cellular to anybody who buys it and you don't have to care which one wins.

You start with product people, but if you do well enough making your product better doesn't really move the needle any more, so organizations tend to promote...

marketing / sales /operations people. These people usually are pretty good at understanding what the customer wants and so has a decent feel for the product, perhaps innovation goes down, but the customer is getting what they want, but then once you saturate the market sales and marketing are no longer going to move the needle so you promote...

Finance people. They usually don't have a great feel for product nor even what the customer wants, but they understand how to increase revenue and decrease costs and at this point in the company lifestyle that is what matters most. The risk is that you are in a competitive space where competitors are willing to jump on any product stumble. Often companies get stuck at this stage and stagnate, but usually they are so large and entrenched they keep doing just fine anyway.

Boeing's previous CEO was an engineer.

One difference I'd point to is that Intel was doing "fine" not committing to future lithography. I put that in quotes because clearly it wasn't a fine plan over the long run, but not spending money on the future is a fine plan in the short/medium term. AMD had been struggling for years and Intel continued to handily beat them. ARM processors weren't a threat at the time either. Intel certainly had the better part of a decade where they weren't committing to future lithography and doing fine.

Before someone says, "but they lost mobile to ARM during that period," lithography isn't why they lost mobile to ARM. Apple was using TSMC's 16nm process in their September 2016 iPhone while Intel started shipping 14nm processors 2 years earlier. Mobile chose ARM when Intel wasn't behind on lithography.

With Google Fiber, not choosing had immediate repercussions. With Intel, the repercussions took the better part of a decade to manifest. Google just decided it didn't really care about the home internet business. No one at Google could say "yea, we're not rolling out wired or wireless home internet and the business is booming." Intel didn't decide that they were exiting the processor business, but their processor business was doing "fine" without this decision being made. Intel could say, "we aren't investing in future lithography and the business is booming anyway. Maybe future lithography is just a big waste of money."

You're correct that not choosing means you lose. However, sometimes it isn't obvious for a while. Google Fiber's lack of decision had obvious, immediate results and you couldn't delude yourself otherwise. Intel could delude itself. Execs could write reports about how they were still ahead of the competition (they were) and how they weren't wasting money on unproven technology. Fast forward a decade and they're not fine, but it took a while for that to manifest.

Plus, if Apple hadn't helped push TSMC forward so much, would Intel be in quite as bad a situation? Qualcomm has been happy to just package together ARM reference designs with their modems and it's really just their poor performance compared to Apple really pushing them forward. While Android users on HN might be buying Snapdragon 8 series processors, the vast majority of Android devices aren't using high-end ARM cores. The vast majority of the market for high-end ARM cores is Apple. If Apple hadn't made a long-term commitment to TSMC for 2016-2021, would TSMC have pushed as hard on EUV? It's a lot easier to invest when you have a guaranteed customer like TSMC had in Apple.

If Apple hadn't pushed performance so strongly, would we have seen as much EUV investment as quickly? It's unlikely it would be pushed by the Android ecosystem where most processors are low-end. TSMC serving Apple meant EUV investment. Once Apple was shipping extremely fast processors, Qualcomm and others wanted to be able to get to at least 50-70% of what Apple was offering (so there were more buyers). Once it was available, AMD could use it to push hard against Intel. Once there were more buyers, Samsung wanted to make sure that its fabrication business was at least in the ballpark.

But if Apple hadn't been focused on taking a strong performance lead, it might have been another 5+ years before Intel's lack of decision came back to haunt it. If it had taken 12-17 years instead of 7-9 years for others to put the screws to Intel, they would have basked in its profits for a long time as its execs were touted as having amazing insight. Of course: you're right. Eventually, Intel would have gotten its comeuppance. But Intel could have pretended it didn't need to invest in the future for a long time. By contrast, when Google didn't make a decision on wireless or wired, that was just the end of expanding that business.

> lithography isn't why they lost mobile to ARM

I'm pretty sure that ARM won mobile because it is much lower power.