Comment by Nextgrid | Hacker News Reader

Comment by Nextgrid | Hacker News Reader

Parent Full thread

If you’ve got 100k people to run something that should run essentially on autopilot, you’ve got much deeper problems where merely laying off selected chunks of people will no longer help. The whole company is rotten and the only way is to start from scratch and not make the mistakes that led you to accumulate 100k people.

I can't even begin to imagine what those 100k people actually do. For starters, none of the telcos actually develop their own equipment - they buy pre-made from vendors like Ericsson. Often that includes ongoing maintenance too. The only "engineering" is building the back-office and customer-facing UIs, and even that is often outsourced (as a rule of thumb, if something can be outsourced, telcos will do it: https://berthub.eu/articles/posts/5g-elephant-in-the-room/).

Customer service might be part of that number (assuming that too isn't outsourced), but even then 100k feels extreme.

10k is ok although leaning on the more bloated side. But 100k?

Telco infra runs on autopilot?

This is really taking the whole "I could build Dropbox in a weekend" style comments to a new level.

It's well known that big tech companies are overstaffed. You probably can't build Dropbox in a weekend starting from scratch, but a smaller scale cloud-based storage solution can be deployed very quickly if you start from existing open source components. And a small team of experienced web devs can certainly build a cloud storage thing in a matter of weeks from scratch too.

> It's well known that big tech companies are overstaffed.

To who? What evidence is there either way?

Is it an often-repeated story online? That sort of information is both well known and unreliable - it's well-known misinformation.

Almost; you need maintenance and monitoring but that doesn't take anywhere near 100k people - assuming they even use their own headcount for this instead of just outsourcing maintenance to their equipment vendor.

Big Tech companies operate much more complex systems (for starters, they actually build greenfield stuff instead of buying ready-made equipment from a vendor and plugging it in) and have way less headcount.

Carrier-level telco is labor intensive.

You're building new cell towers, managing countless failed backhual links (thanks to fiber's natural enemy, the backhoe), working with whatever obscure bugs your MVNOs have managed to uncover, certifying new cell phone designs, and still working on upgrading everything to 5G while simultaneously planning for 6G (keeping in mind that the 5G network architecture looks radically different than the LTE architecture). Much of that work is necessarily physically distributed across the entire country.

Not to mention dealing with end-user sales and support, which unfortunately often needs physical stores.

I'm not going to say whether 100k is too many, but there's a lot more involved here than just maintenance and monitoring - especially if you want your network capacity to keep up with growing demand.

lol. You figured it out, Verizon wireless can just be replaced by a UniFi router and ChatGPT.

ChatGPT can very well be an upgrade compared to the "engineering" capability of a lot of telcos (they have very little, are hell-bent on outsourcing as much as possible and are even proud of that). But don't take it from me, here's a more reliable source: https://berthub.eu/articles/posts/5g-elephant-in-the-room/

VZW has 146m lines. Each employee supports 14,600 customers, seems like a reasonable number...

It does if you have no idea how it works.

Erlang fault-tolerance FTW! ;-)

I feel like vzw abandoned sane things like otp for k8s ai and blockchain ( or whatever nonsense tech goes here ). otp was probably deemed antiquated and too hard to maintain.

Obviously, you know exactly how to run a major telecomm operation ten times as efficiently as the dominant operator in the most prosperous nation on earth — you are wasting your skills and should absolutely be given funding to disrupt them and make Billions!! What is holding you back from joining the Oligarchs?

Have you considered that the inefficiency is a feature and provides cushy jobs for a lot of people and subcontractors? But yes, a modern telco can absolutely be run more efficiently if you operate it like a tech company and don't have to deal with decades of legacy sludge (whether bloated headcounts or heterogenous legacy infrastructure you have to support).

The problem is that this is a culture problem and once a company is ossified it is really hard to enact such change from the inside even if you wanted to because everyone enjoys the status quo (and who doesn't wouldn't be there to begin with).

Another example: have you seen the UK & EU banking scene and the boom of fintech and "neobanks" around 2017 like Revolut, Monzo, Starling, N26, etc? They managed to build from scratch on relatively shoestring budgets their own implementation of a consumer bank, something that their legacy competitors still can't replicate despite having way more budget and resources.

Unfortunately, the telco world is an oligopoly and they don't like new entrants (banking in the UK was actually a much more level playing field in comparison), so we can never actually see an experiment that proves or disproves my theory.

> Have you considered that the inefficiency is a feature and provides cushy jobs for a lot of people and subcontractors?

Yes, this explains most of our jobs.

OK, have you considered that if inefficiency is a key feature and provides cushy jobs for a lot of people and subcontractors, that 1) there would be a low level of average competence in management as well as the workforce, 2) layoffs would have included "buyouts" and RTO mandates that drive away the most competent first so further reduce the average competence, so 3) the layoffs reduction in both aggregate competence and manpower would have been a significant cause of this outage, which is still happening?

I also note you kind of glossed over the 'legacy sludge' technologies that telcos must support — that likely takes a lot of manpower, making it very difficult for them to get to the 0.1x staffing level

These magic devices just teleport out to the location, connect and configure themselves?

The problem with being a nationwide ISP - and Verizon runs mobile phones, fiber and all kinds of other stuff - is that you need lots of hands across the country. A lot of stuff can be done remotely and with automation, but often enough you still need actual physical hands on site, and you can't just say "eh, we'll come around tomorrow, we can't make it there faster".

Also 'Verizon' is not really one company. It is 2. Telco and wireless. Each of those is a mashup of dozens of other phone companies VZ gobbled up over the years. With tech stacks going back decades. At one point while I worked there about 10 years ago they were running the 56k dialup for AOL. They also run a decent amount of stores. They are not going to automate a retail store in the same way you would amazon. You have to have people standing there. Then there is the "i need to talk to someone about why my phone keeps doing weird things" helpdesks/servicedesks. Then the line workers like you point out. Plus the backend people who might be able to work from home. But only if they are not in a secure area working (they have lots of that). That 15k of people was probably the result of several big projects that were scaling up but didnt work out. They have all sorts of projects to try to 'monetize the last mile they own'. Almost all fail.

Are amazon stores automated? I have Amazon fresh store next to me, they have the smart grocery carts that no one uses because they are overly finicky. They have same number of employees as other grocery stores.

That was my point I think I said it badly. A physical store takes people on site to run it. Even amazon with its online store has an army of people running the behind the scenes things. Most things are not automated by code. VZ is not much different. When I worked there they had well over 5000 locations worldwide that were staffed by people. They reduced it at that time. Hence me no longer working there. But sounds like they put it back and then some. Their core business is probably small. But they have huge initiatives they try to do. They do not want to be the commodity data line company. But most of their stuff just doesnt stick well. They think in terms of 'number of lines sold' instead of 'number of customers satisfied'.

Reply on news.ycombinator.com