Saying Goodbye to Louisville
fiber.googleblog.com
fiber.googleblog.com
That's why we can't trust Google with anything critical. They view the world as an experiment they can quit anytime. Definitely don't buy a Google self driving car or anything else you expect to work in a few years.
There is a good chance that businesses moved to Louisville because of Google Fiber and they are being shown the finger "Thanks for indulging us, we got what we wanted and you can go f... yourself".
It would be astonishing if the total number of businesses for which that's true is more than two. I can't imagine what sort of company would base a decision to move to Louisville on the availability of a Google Fiber Internet connection, nor can I imagine how the loss of Google Fiber would be any kind of serious problem (there are other ways to connect to the Internet in Louisville).
I suppose it might be one small factor a developer might put on the "pros" side if her list when deciding where to live, but it's surely a very small factor. Would anyone seriously choose a city to live in based on a marginally faster or cheaper Internet connection? I find it difficult to believe, but maybe it's true for some.
https://k8s.devstats.cncf.io/d/8/company-statistics-by-repos...
Disclosure: CNCF develops DevStats
As far as I know fiber isn't the only provider anywhere, and getting internet set up if you're already wired takes hours or less in most places (I left the Comcast store and was using my wifi an hour later last time I moved).
Eh. Spectrum (formerly TWC, formerly Insight) upped their max speeds. AT&T did a very limited Gigapower rollout to selected areas.
The lasting impact, if anything, will likely be that Google got the metro government to pass a one touch make ready ordinance and defend it in court.
It takes a while to do to those customers.
It can happen as soon as "let's update the webpage" for new customers. Especially when the only high speed internet game in town goes from three providers to two.
Hope I'm wrong.
Those "100% speed increases" were ISP's willing to lose some money to retain existing customers in select markets, using their other profitable markets to subsidize them.
I'm 100% for more competition in the quasi-monopoly ISP arena, however googles entree into the ISP market was just freshman exuberance. They were flush with lots of shareholder money and had young idealistic founders who figured they could "revolutionize the space" because they were smart and cared.
Over the past few years, google has been shutting down all their naive money losing ventures and getting into the serious business of making money (ie. selling your personal info to the highest bidder).
Google clearly is not in this category. They will drop basically anything with very short notice (two months here) if they don't feel like doing it anymore. That may be fine if you're buying a Google phone but that's definitely a problem for anything that can't just be replaced instantly and for (relatively) cheap.
Can you imagine if people who manufacture components for industrial designs behaved like Google does? "We have these subway trains with busted brakes, but it turns out that they were manufactured by Google and they dropped the project two years ago. I guess we have to buy new trains then..."
I can imagine it. It happens every day.
In an industrial setting it is not uncommon to need a part but the stock version of it is not available. It could be that the manufacturer switched from making cooling tower components to fly swatters. Or, it could be that the system you're maintaining is a hundred years old.
You might find a 3rd party replacement part from China. If not, you try to build it from generic components (everyone uses the same bearings, o-rings, springs, etc). If that fails you find a company who sells a similar product and give them your specifications. Finally, failing all other means, you pay a machine shop (or a foundry if you're making the cast iron part of the braking system) to build your part. Anything that has already been built can be built again.
If you want security in terms of products you are buying, you get a warranty. There are laws that enforce warranties. So you get a warranty, and when its going to expire you get an extension, and when you cant renew anymore you plan to replace that product.
The same is true of utility companies. If you are worried about fly by night fiber companies, you should require a lengthy advanced notice-- perhaps a year or two. And if Google blesses some other city instead of yours because you wanted the warranty, then you let it go.
That's the thing. Google uses its big name and reputation to gain entry but they behave like a fly by night operation.
Like in the NYC Subway?
https://www.cbsnews.com/news/mta-why-has-the-nyc-subway-gone...
> The old-fashioned system requires intensive care. When parts break, which at their age happens often, this busy repair shop springs into action. Like doctors, mechanics examine the patients. Some sound like they're on their last breath. Many of the companies that made these components are long gone. So workers here have to manufacture their own replacement parts.
You have to be aware of the potential for the obsolescence of any service you use. Sometimes they drop the project, sometimes they go out of business. The end result is the same. I don't get mad at PageNet for not supporting my pager any more.
Comparing 80 year old parts to a project that is barely two years is not really fair.
Google Fiber was more about scaring ISPs into investing in fiber, than serving its own fiber. Once bribery and lawsuits failed to stop Google Fiber, ISPs started to compete.
https://thenextweb.com/google/2017/07/21/how-google-fiber-wo...
I’m pretty confident that my pest control company will be around for a while. If there’s a problem with my HVAC system, I know I’ll be able to call the people who installed it. The company I bought my cars from will be selling cars for a long time. And I could go on.
Google has the right to do this, but they’re starting to gain a reputation for it and that’s not going to be good for them.
Starting? It's been a running joke around certain communities (including this one) for years and years.
I don't use anything from Google I'd consider critical if there is a reasonable competitor.
Say what you want about Microsoft they generally don't just chop you off at the knee's (I mean they've done it but not in the way google does).
Telco is regulated, so if Verizon abandons your market, they will need to have a plan that leaves you with service. Which is most likely to be selling it to Frontier.
That Google is willing to abandon markets for Fiber is really a testament to their strategy here: Google fiber was not a real attempt to be an ISP, they just wanted to show it was possible to inspire incumbents to get gigabit out to people. Once AT&T started rolling out gigabit fiber, Google was done.
No it doesn't. That pest control company may pack up and leave (although this is admittedly unlikely), but AT&T and Verizon are legally obligated to offer POTS service.
And I’m not talking about POTS service. I don’t have it.
So for example how can I use their e-sim provider Fi if I know for sure in one of my travels I will get a notification 'we are closing Fi next week. Find new provider.'.
I use their Photos service extensively. But in a couple of years I know it will get closed too, once they fully used it's capacities for training internal ML models.
The also got rid of discussion and blog search which along with getting rid of google reader helped accelerate the siloization of the internet.
To be fair, users aren’t customers. People don’t have any right regarding your product if you give them for free.
Google gets this for free.
Yes, you can see Google users aren't customers, they are more like unaware/unpaid workers, providing free tuning service to Google's always hungry data analytic engine.
Google did a remarkably poor job in Louisville, and it's absolutely no surprise to me that they're giving up.
Google's strategy was to build a FTTH network with microtrenching. In practice, this meant cutting a 2" deep groove in city streets, placing a fiber optic cable in it, and using an expanding rubber gasket material to cover the groove.
This failed in spectacular fashion. One of the best pictures was this: https://www.wdrb.com/news/belknap-neighborhood-residents-con...
... a fiber optic cable, barely buried under the surface of the road, exposed after a few freeze/thaw cycles.
If you click on no other link about Google Fiber in Louisville, click on this one. This scene is repeated all over the city.
There won't be much in the way of "dark fiber left behind" because most of the Google Fiber rollout was FTTH done ridiculously poorly.
And for the customers? The only company competing for residential gigabit service is AT&T, which (wisely?) rolled out their service to high density apartment areas and wealthy condos; for most residents, Spectrum or Uverse DSL are the only real competitors.
Some cites:
https://www.techrepublic.com/pictures/photos-how-google-fibe... -- Microtrenching will save Google!!
https://www.wdrb.com/news/google-fiber-announces-plan-to-fix... -- uh oh, things aren't so good...
https://9to5google.com/2019/02/07/google-fiber-closing-in-lo... -- we give up, 2" of trench won't do it.
Key quote:
> We would need to essentially rebuild our entire network in Louisville to provide the great service that Google Fiber is known for, and that’s just not the right business decision for us.
It seems, having gashes in the road, exposed to salt and wear, will increase rate the roads require maintenance. These cities need to monitor and ensure Google is paying those bills.
Read as: We messed around with your street infrastructure using untested construction methods, and it didn't work very well so WE OUT PEACE. You know what they say, fail fast byeee!
It's not a scenario I would have ever planned for, to say the least, but anyone who has service with Google is probably used to it.
I don't know where this expectation comes from that you should be able to pick a service provider and then use them until the end of time. Hell, it's harder to change your primary bank than to change your ISP. Businesses disappear all the time. Every time I move I go through the same hassle of scheduling an installation of new Internet service.
This strikes me as a particularly poor counterexample, as those businesses were effectively fungible: outside of your local ISPs email address, there was nothing keeping you from stopping your service at 718-867-5309 and switching to 718-387-6962. The underlying infrastructure for either of those ISPs was handled by someone else.
Last mile, yes. But there was plenty of ISP specific infrastructure.
Is it though..?
I just changed my primary bank. The most difficult part of it was listening to the account manager person tell me about what a terrible mistake I'm making and how the other bank is just using me.
In some of my previous residences there is only one choice for ISP.
The problem with overbuilding is uptake rates, its not good to only capture 28% of the market or similar, you really need 40+% to break even. The City of Tacoma ran into this with their cable network, and so has Google. Webpass is Google's fix for that, only targetting larger buildings where a 10% uptake rate gets them in the green.
That's not really an acceptable way to operate what is essentially a public utility at this point. We clearly need better regulation of these companies.
Oh word! They are in Louisville? Where do I sign up?
Hint: "average prices" for high speed internet mean very little in the USA as those prices vary wildly. I pay $30 less for 300Mbps service than a remote coworker does for 75Mbps - from the same company.
Most of Centurylink's territory has only halfhearted competition from the local cable company (Comcast, Charter, Wave), and they just recently lost a lawsuit where the state AGs from Oregon, Washington, Minnesota and a few other states ganged up and prosecuted them for hidden fees and malicious billing practices, thus why they are on the straight and narrow now, with the price you see being exactly what you will pay.
I expect Comcast rates will shoot up now that another player has left the field.
Most recently Frontier Communications comes to mind.
Kudos to Google Fiber for actually trying to innovate in a stagnant industry. I'm no insider here, but I suspect the micro-trenching did go through internal testing which looked promising before they approved it for real city use.
If the micro trenching had been successful we'd be cheering for Google for finding an innovative way to bring fiber to more people. Turns out it didn't work in their one trial city and since they discovered this at a time when Google Fiber is no longer expanding it is no surprise that they don't "re-open" Louisville and retrench the entire city.
For a community that celebrates failure as a natural consequence of innovation, I'm surprised at all the negativity here. Not a mention of the 2 months of free gigabit internet they're giving to customers. Yea, Google Fiber is just a bunch of soulless profiteering corporate jackasses /eyeroll/.
The community here isn’t digging on Google trying to innovate. The anger comes from a highly successful company taking a risk, seeing it fail, and just peacing out leaving others to clean up the mess.
If Google Fiber misrepresented their microtrenching then that would certainly be fraud and they would be liable for damages.
When I try a crazy new idea at work, and it fails, I don't just leave a mess and walk away. Come to think of it, I don't do that if it succeeds, either. Supporting your mess is part of professionalism.
This is the key difference between doing research and providing a service. Are you serving the people or just the technology? Google is full of smart people who love technology and want to do research all day, but they're not very service-oriented.
Should they be? Eric Schmidt said of Google Fiber, "It’s actually not an experiment; we’re actually running it as a business". I'm not seeing that here. They learned something valuable about a technology they're using, and they're leaving customers in the lurch.
You make it sound like the people of Louisville are now in a worse position than before Google Fiber came, and that they've somehow left the community in a lurch. All that's really going to happen is the subscribers go back to whatever ISP they had previously.
Now if there's evidence that Google Fiber is abandoning a liability for torn up roads, or misrepresented their micro-trenching to the city council that made the deal, that would definitely qualify as "leaving a mess", but I haven't seen evidence of that.
Is there some sort of institutional ADD?
I work at Google, opinions are my own.
Obviously this is my personal take, but I don't think that's what it is.
To me the issue is the performance/rewards system. Imagine you're someone high up and in charge of 10 projects, all asking for headcount/resources, while you have a limited amount to give. Presumably you're going to give more headcount/resources to the projects that are growing faster. If the differences in growth are substantial, maybe you'll decide to actually just let some projects die so their teams can be absorbed into the other projects with better prospects.
Google's performance system is an attempt at being really objective, and it's hard to put a precise value on goodwill lost by discontinuing a product. Not only that, but the effects of discontinuing a product are felt company-wide as opposed to by an individual. Nobody is going to look up who decided to discontinue a product after all.
I would think this would be a top TGIF question almost every week at this point (because it seems that's how often Google product shutdowns are in the news).
Actually this week (!) was the first time my question was upvoted enough to be asked and it was exactly about this.
My impression is that they are aware but willing to trade our reputation for the ability to experiment or be more aggressive in funding more successful products
Absolutely not. Your presumptions are misplaced.
Assigning resources to growth is usually the most stupid thing to do. By definition the projects that can grow the most are the projects with no customers or revenues, especially new projects that are nowhere, they can only grow. https://xkcd.com/1102/ . Investing in these is to the detriment of established projects that actually bring revenues and keep the lights on, and companies that neglect to keep the lights on tend to die.
Assign resources to the projects where they have the most impact. And business impact should be measured by return on investment (or loss if not invested), this could be almost anything in practice, growing revenues/customers, maintaining existing customers/revenues, not getting fines and lawsuits, reputation control, assets/inventory/cashflow optimizations to save money, etc...
Well you can consider growth in terms of total users and not the percentage.
> Assign resources to the projects where they have the most impact. And business impact should be measured by return on investment (or loss if not invested), this could be almost anything in practice, growing revenues/customers, maintaining existing customers/revenues, not getting fines and lawsuits, reputation control, assets/inventory/cashflow optimizations to save money, etc...
We don't disagree. You are talking about what makes sense for a business while I am referring to what makes sense for an individual in an environment where growth is what determines your performance review. That is a bit simplified obviously since of course revenue does matter.
My point is that it's exactly this misalignment of what's good for the company vs what's good for individuals working there that is the problem.
You can't "disrupt" the power company, and you can't "disrupt" the water company. Not without a giant warchest to rebuild pipes and electric lines. Not without causing a lot of accidental incidental damage. No one actually wants that, they want to pay their utility bills and move on with their lives. It's past time we admitted that internet service is in the same category.
This is a case where to maintain customer trust, they should suck up the cost themselves rather than trying to recover it through the one-time customer fees that they used at rollout time.
hey guys, i found the project manager!
Google Fiber isn't expanding to new cities.
That said, I appreciate them in that they really lit a fire under TWC/Charter/Spectrum. If I remember right, we were 10down/3 up when fiber was announced (possibly 30 down, depending on when). Our base is now 200 down.
...yet.
Just more evidence you cannot rely on Google for long term product outside of search.
...yet
The main issue is that current companies have extreme control over market thanks to regional monopoly. They can squeeze as much of money from users, because there is no real competition. When a new company enters specific region they can up their speeds, lower price (and even operate at a loss in that specific region), so a competitor will have to leave.
The solution to this requires regulation, and basically a law should mandate to allow leasing infrastructure (the fiber) to others, of course not for free, but for a fee.
Currently if we would have great scenario when we would have 10 ISPs available to choose from, we would need 10 fiber cables to every home. That means if someone chooses one, 9 of them are not used, and all the effort to lay the cable and getting permits was just waste of money. If this is expensive to even company that has "unlimited" amounts of money, like Google it's unrealistic to do for everyone else.
Leasing the physical infrastructure is the key to bring competition back in that market.
Google should NEVER be trusted with other infrastructure projects going further. Infrastructure isn't some forprofit companies trial-and-error playground, it is about stability and promise to the local residents, it is not about money and certainly not an experiment.
They've been doing that a lot.
Why? It's just an Internet connection. Customers can call the local cable company and be up and running in a matter of a few days. Might cost a bit more and be somewhat slower, but it's hardly an existential crisis.
Can't really joke about their words that 'the last 2 months are on us', like it is something worthy of cheering up..
From https://cloud.google.com/terms/ :
7.2 Deprecation Policy. Google will announce if it intends to discontinue or make backwards incompatible changes to the Services specified at the URL in the next sentence. Google will use commercially reasonable efforts to continue to operate those Services versions and features identified at https://cloud.google.com/terms/deprecation without these changes for at least one year after that announcement, unless (as Google determines in its reasonable good faith judgment):
(i) required by law or third party relationship (including if there is a change in applicable law or relationship), or
(ii) doing so could create a security risk or substantial economic or material technical burden.
The above policy is the "Deprecation Policy."
Aka, it costs a lot of money, and they don't want to spend it.
So they can easily decide not to stick with it for a year, or they can decide to continue to provide service without, say, any technical support.
Sadly, I think Google's pulling back because it more or less got what it wanted out of the telecoms in terms of network service, repealing net neutrality and regulatory capture increases Google's competitive moats, and because corporate lobbying may be cheaper, more flexible, and have greater returns: https://gizmodo.com/googles-parent-company-spent-more-on-lob...
On that note, here's a how-to on starting your own ISP: https://startyourownisp.com/
In other words, "we fucked up and so we bailed." Google is like the bad first husband you should have never married in the first place.
"not living up to the high standards we set ourselves" is about as helpful as hearing over public transport intercom "there has been an incident".
That's a picture of a road post-fiber install. The groove is where the fiber was placed. That black snaking rubber was what they sealed it with.
Now imagine that, but across their entire service area.
If that's accurate, it's comedic anyone thought that would work.
For a local news source: https://www.wdrb.com/news/belknap-neighborhood-residents-con...
Here's an article proclaiming that this is how Google would "outbuild their rivals": https://www.techrepublic.com/pictures/photos-how-google-fibe...
I cannot possibly oversell just how terrible the work was here. Yes, that is indeed a fiber optic cable two inches under a city street, covered only by expanding foam rubber. Yes, someone really did think "yeah, this'll be just fine."
I guess they eventually said "screw it, we will just pull out"
Since that was only a few months ago, my guess is they decided to just pull out instead.
I can't imagine anyone would want to touch that "dark fiber". It was in terrible shape six weeks after they put it down, before winter hit the city. The picture on this article looks like it was taken relatively soon after installation [2] - I've seen streets where the rubber seal has expanded and curled up over top of the groove that was cut.
[1]: https://9to5google.com/2019/02/07/google-fiber-closing-in-lo...
[2]: https://www.wdrb.com/news/google-fiber-announces-plan-to-fix...
Seems horrible if Google tore up roads to put in Fiber and now they decide to pull out leaving the taxpayers holding the bag.
I wonder if a situation like this was in the contract with the city. Maybe a journalist will FOIA them for a copy, but something other cities should think about when they do public-private partnerships - what happens when either partner doesn't want to be involved anymore.
The sober light of day makes it pretty clear that Google was attempting a pilot of a very certain type of installation that failed spectacularly.
When it gets cold enough, the water in the ground freezes and expands and basically "squeezes" the fiber. The "1"'s, which are long and skinny, can still slide right through the fiber but the fatter and wider "0"'s sometimes get stuck. This causes the fiber (Internet pipe) to get clogged and, eventually, stopped up completely, similar to how car crashes on the Interstate can quickly bring vehicular traffic to a standstill.
(Source: I was the Senior Network Engineer for a FTTH ISP, until just recently.)
EDIT: As others have said, I think the problems in Louisville come more from shoddy installations thus making them more susceptible to extreme weather rather than 'fiber not working in cold weather'.
I just don't understand why they didn't spend more time testing it on their own resources before rolling out to actual customers in a major metro area. We've all seen potholes, roots coming through streets and sidewalks. Just crazy to think they thought this was ready to be deployed at scale...
This is just irresponsible of Google at this point. This entire project has no momentum and isn't going anywhere -- why are they still signing people up when it's obvious that they'll just cut them off eventually as well? In the meantime, they prevent any of these cities from taking control of their own connectivity.
They should help the cities form nonprofits to run the fiber networks, then gently remove themselves from the situation.
They're still "microtrenching" in other markets but at a much greater depth (6in rather than <3in) and with a different cover strategy (asphalt replacement over sealant rather than sealant only)
They were experimenting with real customers, as google often does. Only this time it was what many would consider essential infrastructure.
After "Advancing our Amazing Bet", it became clear that the level of regulatory obstacles Google needed to overcome made it too cost-prohibitive to continue their rollouts. But what factors are forcing them to pull out of an already established market?
Is the ISP environment so hostile as to squeeze even the biggest players out of town? Is Google slowly ramping down even existing Fiber operations? What's going on behind the scenes here? If the microtrenching issue is driving this, why can't Google of all companies put up the investment to re-do it right?
There's failing an experiment, and then there's realizing something is sub-par and putting in the work and money to make it right for your customers. I feel like Google has misconstrued the two here.