Dear Google Cloud: Your Deprecation Policy Is Killing You
medium.com
medium.com
The reason is the engineering culture and the promotion system. People expect to switch to new projects, and work towards to a promotion within 2-3 years. This shows in all products, and presumalby also the reason for the deprecation. New people come on the project, want and need to do a major overhaul, and do not have resource or incentives to support the old stuff.
The other half is that most projects are done efficiently by relatively small groups of engineers. I remember that Android Market was still a few people in MTV, with no payment possibilities outside of the US, when Apple already had significant revenue from apps. And that shows everywhere. I'd argue Google is incapable of tackling projects that require more than a few hundred people for more than 2-3 years.
But as a billion dollar company, that does not work. Throwing money at obvious new billion dollar opportunities like Microsoft did with gaming and cloud, and following through over 5 years is not possible for Google.
Looks like someone deserves a bonus
You need the people evaluating performance to really understand the work, and make informed, discretionary decisions about who to promote to maximize real value creation.
I agree but for everyone working at / running a company without a money printing machine it is worth noting that this is really expensive. You need to take someone that’s a good engineer—-one with judgment and a modicum of people skills—-and give them mostly thankless, stressful work.
If you start rewarding maintaining old things you aren't going to suddenly get what you want, you're just going to lose a tonne of people because they don't want to do what you're telling them and you won't be able to hire good people with your priorities because no one is going to believe they can have a good career at Google by prioritizing stability and long term support. It's all downstream of culture, and culture change is very difficult.
Of course if you do buy into this idea that this is the issue, then GCP has already lost. It would be years before they could make a dent in that culture, and more years after that for external customers to know and trust Google's new culture. By that time there's going to be a dominant player anyway. So maybe actually it's best to stick with the current culture - try and win on your strengths than try to fix your defficiencies.
Keeping backward-compatibility requires people that care. That have enough pride in their work to counter-balance the grind.
It might not be a good business decision for Google, but surely if they create enough positions that are solely focussed on the boring stuff but are paid better to make up for it, there is a price point at which they'd get enough interested people.
MS pay is actually competitive for external hires, although internal raises aren't usually very high.
This is true for most people working for large multinational corporations. Good things can still be developed without passion.
There's nothing wrong with that, and when I'm at work, I do my best to do a good job and make reliable, maintainable systems. Those two things aren't in conflict.
Isn't that inherent to their interview process?
I don't follow. Wouldn't ridding Google of its reputation for inadequate maintenance, make it a more prestigious employer? Its reputation for paying well wouldn't change. How would this lead to an exodus?
Obviously generalizing here, but most engineers (especially fresh graduates) would rather work on a new thing than do maintenance, if given the choice. Google offers a pretty good value proposition: Lots of money and you get to work at new exiting projects and if you don‘t like it you can change to something you like. There‘s obviously more nuance to this, but that seems like a pretty good deal for many people.
I think that's only part of it though. I think that the only way a company can let products wither for years without a single new feature or improvement is if they don't have a product manager to represent the users, and bring a vision to the engineering team.
This is the same reason why it is near impossible to reform a police department. It would take firing everyone and starting over.
Damn, I was waiting for this. The whole essay was a setup for this paragraph. Reframes the argument using a shared traumatic experience for everyone. This humanizes the effect, we all know someone that lost someone to the Python2/3 transition. I can't count the number of Python friends that turned into Gophers, joining some cult of middle class bearded tech dad with a side business distilling pear brandy.
Best Yegge Essay Yet! Welcome back Steve.
edit, finished the essay. He smoked GCP in this one, damn damn hard. Hickory.
Of course the language wasn’t as broadly used back then and Apple is known for quicker churn than Python so the need to write code compatible across Swift versions was much less pronounced
The back-compat promise is one of the reasons I'm investing in Rust for new projects, even if I find the actual programming much more pleasant in Swift.
But I'm a bit less optimistic about Rust now given the recent layoffs at Mozilla.
I'm a bit surprised that language support for such a key concurrency feature comes so late in the game.
It's not the same issue that Yegge describes though.
Async/await does not need to be a breaking change - C# managed fine without it being so. The breaking changes in Rust were around the standardisation of std::futures::Future rather than async and await.
That said async/await makes a world of difference to the usability of Rust for async code and I’d imagine it will for Swift too.
The problem isn’t making an incompatible “v2”. It’s getting rid of “v1”.
If I can’t take advantage of new functionality without moving to a new version that’s a headache. But it doesn’t cause me to throw away all of my existing investment. Especially if you can combine the old and the new.
Flutter and React Native both work on iOS. And obviously Objective C still works
As an example core of Unity is written with C++ with Objective-C bindings for APIs such as Metal.
If Apple would mandate Swift then things like Unity would just cross compile into swift, causing perf regressions for users, all for no benefit for Apple.
This feels like a very strange perspective. Why is it a problem to "lose" someone from a language? You're an engineer (a presumption on my side); casting yourself into a subsection of a "Python engineer" seems like a move with all the downsides and no upsides.
I dread the day Python will be as bloated as C++ or Java, neither that dare to remove things.
Arguably worth it. Dividing 1 by 2 with "1 / 2" and getting 0 is a real 'wat' for novice programmers. And the workaround is a 'wat' for experts: instead of using a floating point division operator, cast one of your parameters to float.
Joking mostly aside, python is way way closer to C++ in terms of language feature bloat; where Java is notoriously spartan and slow moving (moreso historically).
Ofcourse I know you're also reffering to standard lib gunk but that's relatively less impactful.
The big issue with the python 3 shift was when you touch something as substantial, subtle, and pervasive as strings you're going to cause problems. I wont speak to if it was worth it.
Also the python 2 string type was more in line with how other unix utilities work, so while I do think the 3 string type is generally better for the web, it wasn't an obvious improvement for the other large usage of python. While it's a very versatile language, I would say the major niches of python are: web dev, shell scripting, and scientific computing. The changes in python 3 were (somewhat) helpful for web dev, but really were inconsequential and/or actively harmful for the other uses.
Mixed with how condescending the python dev team acted about it, and how it really wasn't until about python 3.6 that there was any compelling reason to move to python 3,you can see where most of the hate comes from.
And by the way, even if Python is very popular today, I do think that rift really hurt the language. I worked at least at one company where Python was considered for a project, but eventually shelved because of management's confusion over the 2/3 issue.
> The "print" statement being a great example. Was it kind of weird? Sure.
I am reminded of the Emerson quip "A foolish consistency is the hobgoblin of little minds". The Py2 print statement is special/weird because it is a very important and special behavior, especially for newcomers. It's not randomly special, it's special because it models something that deserves special treatment.
Py3 is more consistent, but less humane. I use it over Py2 only because the market has moved on, but every time I type print with parentheses I curse Guido.
Can you expand on this, why is this more humane? And why should print be a special function?
print "hello world"
vs
print ("hello world")
Larry Wall of Perl thought about these things in terms of Huffman Coding, i.e. the most common thing you might want to write should have the syntax elide any easily understood context.
In the case of "print", we know we're not talking about any random function here: it's the function for getting output to the terminal, one of the most fundamental operations in all of programming. So I think it's totally fine for it to have a special representation in the language.
One nit I would call-out is that most people don't put a space between the print and the ("hello world"), so idiomatically when learning, print is 'just another function'. And since HelloWorld is usually the first thing people learn, I wouldn't want them to think print is different than any other function.
Also, the first line of The Zen of Python is "Beautiful is better than ugly", I can understand why consistency was a priority here.
We could get into a whole other discussion about whether or not a programming language should have and adhere to a "culture", but fundamentally I agree with the arguments against Python making this change. It's just not worth the effort, and like others have said, they might've lost of a lot of momentum to JS.
That the transition took nearly a decade (and for some still goes at this point) and several minor-versions is another hint at strong problems in planing. At the end we got something better, but the road toward it was just far to painful for what we got. And this fuels hate till today.
Yet Python 2 to 3 started in 2008 - 12 years ago, and is still not “done” in any meaningful sense.
If I think back to 12 years after the intel OSX transition - 2018 - I can’t think of a single outstanding issue, whereas I bash into python 2 vs 3 literally every time I touch it (and therefore mostly choose not to).
I wonder when/if the software industry will ever stabilise. It's possibly the only industry where frequent and disruptive change continues to happen, and is even welcomed by many in it (not me).
> I’ll also give a shout-out to our friends in the Operating Systems business: Windows, Linux, NOT APPLE FUCK YOU APPLE, FreeBSD, and so on, for doing such a great job of backwards compatibility on their successful platforms.
I share your desire for stability. I don't necessarily mind frequent updates for security reasons, but I'm getting to the point where if things are changing in big ways often, I'm just not going to use it --- I no longer find enjoyment in running on the upgrade treadmill, it's just work.
That said, I don't see a lot of hope for our outlook. Most people don't seem to have a concept of completable software. It would be nice if it were bug free, but getting to the point where the cost to fix bugs compares to the cost of leaving bugs as known issues is doable.
This should be generally true of any well-compiled static binary on linux from 20 years ago.
No way! IBM mainframe (zSeries) is the gold standard for backward compatibility in IT.
Your Win32 binaries sound better than whatever software is being distributed by the companies today.
See also my other comment[1] - Angular is an example of such a magical does-everything framework.
Angular is the only one that has remained even remotely usable, but certainly not stable. It's never as simple as running the upgrade utilities, changing the version, and being done. It ALWAYS takes at least a day to find all the "little" things they didn't feel fit to mention in the upgrade docs.
It's aggravating. I genuinely don't like using Google software, at all.
I've felt for a long time that Google is coasting on the momentum of the early web and the awe people felt for what they built early on. That hasn't been Google for a long, long time.
I really like Node as a language (especially with TS), but Node as an ecosystem feels like a very hard fit into 90%+ of corporations.
Consequently, JavaScript developers got used to the burden of maintenance and rapidly updating to the latest versions of libraries, and this culture carried through to Node (partly because a lot of libraries are shared between node and the web).
Having said that, if you pick your libraries well, it's not too bad these days. When I upgraded from Node 12 to Node 14 earlier this year, I had to upgrade the `pg` package to a newer version that supported Node 14 (there was one available), but I didn't have to make any code changes. And other than that I've had no forced version upgrades in a long time.
I guess if you're lookig at 5-10 year timescales with literaly no maintenance then this would be a different matter though.
This is true for any language/platform.
With Clojure, I can think of 2 times ever when a dependency caused an issue. It was extremely obvious since the issue was "won't compile", and the fixes were simple.
With PHP, I expect any change to potentially break something. Bump your AWS SDK which uses a different minor version of guzzle? Fatal error.
There's a world of difference between breakage being an everyday thing and a true rarity.
The short answer: if you avoid the shiny tools that claim to do everything, you will not have this problem. And as a special case, avoid Gulp, which is mismanaged.
The single-responsibility libraries that have a well-defined scope, on the other hand, have often been sitting on npm unchanged for 5-6 years, because they are simply done, and they do what they need to. They will very likely never deprecate anything, as there is simply nothing to change.
I would argue that these libraries are actually doing a better job of stability than their counterparts in other languages.
Edit: An additional factor is that it's easy to write a tutorial about an extensive framework with a large scope; there's plenty of stuff to write about. Writing a tutorial about "this is how you make a function call to this library to parse a geo URI", on the other hand, probably isn't going to happen.
So if you follow third-party tutorials, you are naturally going to end up at the packages that are most prone to deprecations.
It's a long rant with a bunch of f-words, and the claim that he gets deprecation e-mails "about once a month".
Can someone who is informed point to actual GCP services that have been, or are being, deprecated?
From the article, I have no idea whatsoever if this is some huge actual problem with core services... or pulling support for ancient versions of Linux... or sunsetting experimental/beta services that never had guarantees to begin with.
I'd find it pretty surprising for a platform as relatively new as GCP to already be deprecating anything meaningful, so I'd really appreciate if anyone has actual facts here.
(Also I know a lot of people who use GCP, and I've never heard anyone complaining about services being deprecated, so to hear someone complaining about it happening monthly is pretty surprising.)
> I know I haven’t gone into a lot of specific details about GCP’s deprecations. I can tell you that virtually everything I’ve used, from networking (legacy to VPC) to storage (Cloud SQL v1 to v2) to Firebase (now Firestore with a totally different API) to App Engine (don’t even get me started) to Cloud Endpoints to… I dunno, everything, has forced me to rewrite it all after at most 2–3 years, and they never automate it for you, and often there is no documented migration path at all. It’s just crickets.
Similarly, there was tooling released to upgrade Cloud SQL versions:
> In 2016, we started offering Second Generation instances to Cloud SQL customers. Second Generation instances offer improved performance, availability, and storage capacity. In October, 2018, we released a tool for upgrading First Generation instances to Second Generation. On January 29, 2019, we deprecated First Generation instances in favor of Second Generation.
I think maybe Steve's past experiences from within Google are causing him to project the same behaviors onto GCP products, when it doesn't seem to be the case. Though, admittedly I did get bit by the Python 2 issue as well.
I have a friend who built an AppEngine app within the first year of the platform's release, and he's spent the better part of the past 9 months doing almost nothing but managing his company's migration to Python3 AppEngine.
It wasn't just migrating from Python2 to Python3 syntax. Google used it as an opportunity to also turn down tons of APIs and services they no longer felt like maintaining.
One big difference was the Users API. Python2 AppEngine had a pretty simple way to authenticate users that you could set up in an hour.[0] In Python3, they've dumped that entirely and tell you that you have to move to a totally different solution, and they offer no help with that migration.[1]
In Python2 AppEngine, you essentially got a memcache for free. The ndb library[2] allowed applications to read/write to Cloud Datastore, but it also automagically managed the cache for you so that you didn't have to set up a separate cache service. In Python3, they've dumped that and force you to maintain your own cache service and handle all the caching logic yourself.[3]
It looks like the ndb library is pretty complete at this point,[4] but when I was looking at a migration earlier this year, the ndb library wasn't even out of beta. Meanwhile, they were breathing down their customers' necks about moving off the Python2 version.
I had a fairly small (~8 KLOC) Python2 AppEngine app that I wrote in 2018, but when I realized how much of a hassle it would be to migrate to Python3, I just rewrote it using Gridsome and moved off of AppEngine entirely.[5]
[0] https://cloud.google.com/appengine/docs/standard/python/user...
[1] https://cloud.google.com/appengine/docs/standard/python/migr...
[2] https://cloud.google.com/appengine/docs/standard/python/ndb
[3] https://cloud.google.com/appengine/docs/standard/python3/mig...
[4] https://googleapis.dev/python/python-ndb/latest/index.html
I have a ~60 KLOC AppEngine Python 2 app (started in 2011), but I'll probably migrate that over only when the deprecation actually occurs, as:
- that gives the python-ndb library more time to mature (performance and bugs),
- easier migration paths for some of the non-py3 services I use (e.g. Images, Users, Memcache) might appear when deprecation actually occurs, either from Google or 3rd parties,
- I'm in no hurry,
- I will be surprised if Google gives less than 1 year of deprecation notice (Python 2.5 got 4 years from deprecation to termination).
(If the APIs I use were available for py3, I would've probably already migrated.)
Remote build execution - deprecated. Our live systems went down because of it and they didn’t even send an email they were killing it. Some person just pressed the kill switch and even support was unaware.
Their gcloud stuff asks for upgrades all the time. Older gcloud sometimes breaks. Sometimes upgrading to newer gcloud breaks existing code that interfaces with it.
It’s not that bad, but yeah GCP doesn’t give a shit a bit backwards compat as the other players do.
You get the feeling, they don’t dogfood their own shit. Like their pricing calculator is unusable on iOS. It baffles me that their support says “there is a feedback button” but there is no feedback button. They seem to only care about chrome with their cloud.google.com interface.
It really makes me think “what were they thinking?”
However, I do generally agree about the rant. While datastore still works, I can't upgrade my node-js app because the newer node bindings for GCP break the api's, and I can't upgrade my app's Node version either because They decided to ship static WebRTC binaries for the node GCP packages and those arn't avaiable for node 12.x (for the old package version I'm using)
I've had similar experiences with broken Bitnami images that makes it seem like they aren't even tested before being released [1][2]. Their WordPress images do work, but the way file permissions are configured doesn't play well with the way they have WordPress set up [3]. Have I just had bad luck, or is this a common experience?
[1] https://community.bitnami.com/t/cant-authenticate-to-fauxton... [2] https://community.bitnami.com/t/couchdb-server-log-gives-wro... [3] https://github.com/WP2Static/wp2static/issues/598
Can't get too upset, we don't pay them anything. Not in a rush to use them again though.
EDIT Disclaimer: I work for Bitnami; not in that team though. I reached out internally to see if this is a known issue, but it's Saturday...
EDIT 2: does the downvote mean bitnami community support is so bad or did my wording offend anybody?
That said, my take (and that of siblings) is that I'm much better off building my own containers than serving as unpaid Q&A for Bitnami. It would be smart to use containers built by people more expert than I am. Containers that are broken by default do not meet that criteria. So, "reach out to community support" is not helpful to people that have already decided to ditch Bitnami.
> We're careful to pin versions, and hadn't anticipated someone would re-write history and break published code.
This is exactly the kind of thing that will make me ditch a company forever and never look back.
We use immutable tags like "4.2.8-debian-10-r50" which we never overwrite.
Then we have "semantic versioned" moving tags, like "4.2.8", "4.2", ... which resolve to the latest image matching that prefix.
Moving tags are what they are; I'm personally not a fan; all major popular images have that (https://hub.docker.com/_/golang, https://hub.docker.com/_/python, ...)
You can read more about Bitnami tagging scheme at: https://docs.bitnami.com/tutorials/understand-rolling-tags-c...
If you want to pin your image you should either use the most specific tag or just use the digest:
$ crane digest bitnami/mongodb:4.2
sha256:8650c2d92eea97732eae359a140ee86ee3923a2a19b19443e1dc01ec20d5387d
$ docker run bitnami/mongodb:4.2@sha256:8650c2d92eea97732eae359a140ee86ee3923a2a19b19443e1dc01ec20d5387d
> > We're careful to pin versions, and hadn't anticipated someone would re-write history and break published code.
> This is exactly the kind of thing that will make me ditch a company forever and never look back.
On the other hand, assuming that instead of a misunderstanding, what you saw is actually the image behind an immutable tag such as 4.2.8-debian-10-r50 being replaced, this is a serious security issue; somebody could have hacked docker hub, or crafted a valid certificate for docker hub and MitM'd you,
I'd also ditch a company forever and never look back if they honestly don't care about that problem, which I assure you it's not the case.
We'd greatly appreciate reporting such cases to security@bitnami.com / security@vmware.com .
In fact at this point it's easier to use AWS Lightsail.
Buy things from Google, though? You mean, give Google money and in exchange expect them to adhere to some kind of standard of behavior, support, and customer service? Go get a coffee because you're clearly not awake yet.
I do believe that there is a level of spending where you can actually get a hold of someone competent who can look into things. But that level appears to be rather high.
After a year, HP dumped GSuite, and went to O365 instead. We got a responsive TAM, full multi-tier support, and white glove handholding for migration. And, I think it was cheaper, too.
There is NO level of spending as a customer at which Google actually gives a shit.
https://en.wikipedia.org/wiki/Category:Discontinued_Google_s...
SimpleDB still works, it's not even deprecated or grandfathered to not accept new users. There is really nothing else you need to know about how AWS treats deprecation than that single example.
Deprecation is practically unheard of in AWS. If it does happen you can be damn sure it takes years to actually remove obsolete functionality and there is a fairly easy migration path to something of equivalent functionality.
Also: classic vs application load balancers - especially the health checks. That one drove me nuts for weeks!
AWS never seems to "deprecate" anything, but they do enjoy adding services which are very similar to other services but guaranteed to trip people (ie: me) up when it comes to the buried-in-the-documentation details.
"The EC2-Classic platform was introduced in the original release of Amazon EC2. If you created your AWS account after 2013-12-04, it does not support EC2-Classic."
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-clas...
That was 4 years after VPC was launched. So yes, while they do deprecate stuff on the rarest of occasions, they still give you much, much more time than GCP ever did.
Also, EC2-Classic is still available if you reach out to your TAM with a really good reason.
Initially the (justified) change gave you ~18 months to implement necessary changes. Seems feedback was negative regardless, so they modified it in a way so current usage remains unaffected.
Thousands of engineers chasing promotions by dropping support for live code. If it was their code they wouldn't do it. The org is broken. If you want to see what mature software support looks like, check out Microsoft. Win32 binaries I wrote in college still run on Win 10. Google looks unimpressive by comparison. But they all got promoted!
I guess you can't just set it and forget it on Google Cloud either even though everything was legit fine at time of deployment; I mean come on, it's a simple Python Rest API, what you want me to do 7 years later? xD
I remember that same old blog post also lamented overhearing a conversation in which a developer bragged about how their idea for a change was so good that everyone agreed to break the library API to implement it. This reminds me of how a startup being 'highly disruptive' is sometimes fetishised as an end in itself, but of course it's even worse than that: they're celebrating undermining the library's dependability.
Too often it’s “a user error”.. no, it’s a platform/developer/provider ak lower level error or mark of laziness
I stopped publishing own apps on the Play Store, and unpublished all the one that I had, because of the stress and the time used just to follow the damn policies.
They're adding new requirements one has to follow in order to get access to certain permissions.
They're updating their content rules which make anything with user content very very difficult and might just kill your app one day.
They're updating graphics and icon requirements rather often.
It's a rather big amount of work if your app isn't super simple and relies on functionality they change.
Software makes the fabric between everything in our lifes. It brought us tones of innovation; It is one of the biggest enabler, innovation drivers and tool we have. It is one of the complexest and cheapest tools we have.
You know what happens with software which is stoped being worked on?
Very old, unflexible Cobolt Software on Mainframe systems in banking systems. Guess why your bank is so antiinnovation?
Security holes
'Legacy hell'
We just need to do less Software and better Software.
Versioning and backwards compatibility costs a ton of money and blocks your innovation.
Legacy hell comes often enough also from dependencies you can't get rid of anymore...
Deprecated APIs, functions, etc, in many orgs hang around forever. They might not be ideal, but they still work for code written before (or even code written after that might have some perverse reason).
Google (outside of Android) doesn't deprecate, they rip it out. It makes things easier for them, as this post details at length, but it makes it harder for the rest of the universe.
Hell they said they were dropping support for VB6 in Windows 8 if I remember correctly, but I'm pretty damn sure I can still install the IDE, I know VB6 apps aren't broken for darn sure since I've re-downloaded some old ones I used to use.
I was exposed to this firsthand yesterday when I cloned the [ShareX](https://github.com/ShareX/ShareX) project to take a look at a problem I was encountering.
Literally all I had to do was install Visual Studio Community, click the “Open in Visual Studio” button on Github, wait a few minutes to install the C# runtime that Visual Studio automatically prompted me to install, and click “Run”.
5 minutes was all it took to be writing code for that project and it makes me want to get into the .Net ecosystem more.
Project code did not change much since 2010. It even has bits of IE 9 support.
It’s not entirely comparable, but Google is basically in their “Steve Balmer”-phase.
Hopefully they can eventually get someone with competence and actual domain knowledge, and not some asshole who’s been taught they can lead whatever because they’re been “educated” to lead..
On a side note I think this is what all of us should be aiming for. Every time someone on a team I'm working for starts trying to use some very complex tool that a SaaS/cloud has built for us I always urged them to look for a simpler solution.
Do you really need AWS Lambda/kinesis/etc or can you write your code and deploy it onto our cluster as a normal service/api/control loop?
If you make it really easy to write code, deploy it, and monitor it, you'll often find it's way less devops time to write your software as apis/control loops than it is to use a large amalgamation of cloud services. I also haven't found a good way to unit test cloud services and their interlinking.
I want full in vendor lock-in, it saves an immense amount of time.
You just need to pick a vendor that gives a shit about it's customers to be locked in to - and that's simply not google.
However you can take steps that even when you give in to lock in, you structure your code/projects so that switching to another standard is straightforward: include non-vendor build scripts that would work on any Docker/VPS/baremetal server, isolate the locked-in config and scripts from the rest of the app as much as possible (even if it means running separate external scripts to execute them), etc.
Once you've done it a couple times you build up a suite of small tools and it gets easier.
I got bit hard by Zeit/Vercel, and then by GCP, and then by AWS. All of them do one thing or another to manage to find a way to extract more money. My goal is to make it so that if one day I decide we simply need to leave the platform, we can at a minimum of effort.
Because this actually does happen 1-2 times a year, thanks to exactly the issues raised by OP, it's paid dividends to put in the small time it takes to keep things mostly agnostic on the code side.
Network effects and increasing anti-competitive practices will keep Google at the top for years to come, but without a cultural reinvention it cannot possibly regain the prestige it once held as an institution.
[1] https://medium.com/@samo.burja/live-versus-dead-players-2b24...
Things they've removed have included the entire original ActiveRecord query API, and the RJS templating system for Javascript. Those two required rewrites of significant portions of any app that tried to bridge the transition using only supported APIs. And every new point release (say, 5.1 to 5.2) brings a enough more minor deprecations to make an upgrade not a task, but a project. That's forcing a lot of significant rework, and every one of those gives people a chance to just take their project elsewhere.
(There's actually a third alternative. The release after these two major components were removed from core Rails, they were still semi-supported as add-on libraries. In both cases, core support for those was ultimately cut off -- but people with old apps could try to keep at least portions of them working to minimize the damage. I've done that to help avoid rewrites in components of an older Rails app. I didn't support all of either old API -- for example, rewriting uses of deprecated options to `belongs_to` was easier than keeping them on life support. But on balance, it was easier than the rewrites.)
When GCP came out I wanted to switch from AWS to GCP because AWS was costing me a lot of money and GCP was marketed as being cheaper. So I went to their website and signed up. Or at least I tried to sign up, because GCP did not care about individuals, you had to be a company! No other cloud vendor I tested had this requirement! So up to this date I have never tried to sign up with GCP again.
1) I don't want them closing my account. Doing development is the sort of thing which looks A LOT like suspicious activity, for Google's buggy anomalous behavior detectors.
2) They close accounts on working businesses all the time.
But map() (in either language) has the semantics of applying a transformation to each item in a collection to produce a new collection, so you wouldn’t really want to use map() to call something like doStuff(). That's what normal loops and forEach() are for. If you aren’t transforming the collection into a new collection, map() is the wrong choice; you’re telling anybody reading it that you’re doing something that you aren’t.
But assuming doStuff() just has a bad name and is actually transformational, then the most direct translation would be to use map() and a lambda:
new_array = map(lambda e: e.doStuff(), array)
new_array = [e.doStuff() for e in array]Yes. The combination of a botched transition with a high arrogance level was hell. Here's something I wrote in 2015 after porting a medium-sized system from Python 2 to Python 3.[1] I listed a number of package problems encountered during the conversion. Ones that indicated those libraries weren't being used much in production. I discovered that the SQL connector broke if you loaded a database of tens of thousands of records, for example. And oh, did I get hate mail. You can see the comments below mine.
That's what led me to Go. Go is a mediocre language. That's a strength when you just need to get something done. It comes with a set of libraries which are heavily used internally within Google. So, if you're doing something that is server-side for some web-related task, the libraries for that are probably both present and well-debugged. And they don't seem to be deprecated rapidly. I'm not using them for any Google-specific services, so I can't speak to that.
[1] https://lists.archive.carbon60.com/python/python/1187081/?pa...
I've worked the most with AWS so I'll use it as an example, but it seems like they sell on "new services", so they are incentivized to release quickly, which leads to services with a lot of holes and poor documentation, making it almost essential to get paid support to work around these issues.
It makes me wonder what it would look like to build a minimal cloud product which focused on a few core services with an emphasis on reliability, performance and developer ergonomics.
DigitalOcean, Linode, heck even OVH now have K8s offerings with ready-to-go (and free) control node/panel instances.
Of course K8s isn't shallow but at least when you master their way of doing things you're free to go and do whatever you want.
It's cheaper too.
For one example, in most managed k8s setups, you can't change the admission controllers used on a cluster unless the cloud provider exposes this via an interface. Some cloud providers do more of this than others.
So with a minimal k8s managed option you might get stuck if your cloud provider doesn't support an admission controller or other API server option you need.
If your deployment is nothing more than scp from the post-build products on Github, you can use literally any server anywhere with SSH access. No lock in.
But if your deployment is a button click from inside a DO managed service, well... you're SOL if you want to move. You have to build all the stuff you avoided building before (the time savings and therefore money savings).
I really feel like some combination of the two should be possible. I've not yet seen a host come along that offers it. No large company would because they want reliable revenue generated due to lock in.
I've thought about building it. The older I get the more it interests me. I've got all the basic scripts to make it work. Maybe I'll dive in one of these days..
Google flagrantly, with absolutely the “fuuuck you” vibe Yegge writes about, does not care. It fundamentally doesn’t support you. It promises X, delivers incompatible Y, deprecates Y for Z, doesn’t document anything along the way, has no useful support docs, crams upselling ads into all its docs, it’s bonkers.
AWS can be hard to use and confusing. That’s just not the same thing. Poor ergonomics I can forgive, especially if there’s docs and support.
Maybe GCP should take a hard look at their internal incentives?
Most NPM packages.
> Most NPM packages
NPM packages tend to be free and open source so morally they don’t owe us anything. So yes what is important is the time of the authors of the packages. I am grateful someone took the trouble to release them and many popular ones are actually well maintained.
But cloud services that you are paying for are a totally different ball game. The company does need to maintain backwards compatibility for a good amount time and they need to value your time.
Other ecosystems (like Python) don't have the same culture as NPM. There are clean, well-documented, maintained packages.
Google doesn't have a moral obligation to support GCE customers either. It's just that everyone I know who has used GCE would never use it in production, and loudly advises everyone they know not to do so either. They'll be bleeding money at some point, not to mention they're already bleeding reputation. Google went from smartest-people-in-the-room to smug-incompetent-arrogant-douchebags. That's hard to get over. That's about good business, not some kind of moral obligation.
Almost every day there is an automated PR to bump one of the thousands of dependencies. LTS is ending for Angular 8, but one of the main component dependencies doesn't support 9 yet. I had to put typescript in an unsupported state so one of the dependency upgrades wouldn't break something -- the solution is an ugly hack not supported by anyone. Another required a bump to lodash, which suddenly started failing. This led me to the following comment in a github thread:
> The recursive implementation of cloneDeep was by design with the shortcoming that in some rare cases it could run into stack size exceeded issues. I'm not how cloneDeep is being used but chances are a more specialized clone tailored to their scenario will be a better option.
So, I agree with the OP: the messaging for support in the npm world can be distilled to "fuck you".
If I had ever proposed to a customer that we would build infrastructure for them written by thousands of pseudo-anonymous entities who may or may not break everything because they felt like it, I would have correctly been fired.
This leads me to believe that supportability, reliability, and security aren't important to the npm ecosystem because ultimately the products built with npm don't matter. A bank or a utility company may use them to build a customer facing application, but for the stuff that's important, a competent technology manager would never let npm through the door.
(edit: I replaced "tools" with "software" in the first paragraph, because tools don't change interfaces and effects on a whim).
The article was worth a read for that quote alone.
The list goes on and on.
And yes. Despite these being minor version updates, stuff will break if you go from, say, version 2.9 to version 2.11.
I can tell you that virtually everything I’ve used, from networking (legacy to VPC) to storage (Cloud SQL v1 to v2) to Firebase (now Firestore with a totally different API) to App Engine (don’t even get me started) to Cloud Endpoints to... I dunno, _everything_, has forced me to rewrite it all after at most 2–3 years.
People assumed that because we made a new database we were killing the older one, but that was never true.
But also idk what else you'd want them to say "the thing you're taking shit isn't deprecated" seems like an important response to "you mishandled this deprecation".
Personally, I think most of them seem quite sensible. Having a 'support everything forever' approach is obviously going to impose a huge burden on the teams who maintain this stuff which is then going to limit the ability to make anything better. The depreciation notices generally seem pretty good (12-15 months notice by the looks of it, sometimes followed by degraded functionality rather than complete removal of the feature).
And yet AWS is doing just that while innovating at the same time.
[1] "Depreciate" is something else.
Come to think of it, the Channels API shut down was a bit of a nuisance. But it never worked all that well, and deprecation seemed like a reasonable move to me.
Here's a detailed list[1].
There's a reason that people trust Amazon with their compute, and that's because in regards to their technology they're trustworthy.
Example: https://cloud.google.com/stackdriver/docs/deprecations
You aren't a customer, you're a user who paid to bypass a rate limit. There is a difference.
I mean, there's stuff like Python 2 support being sunset on managed services, but as Yegge himself points out that's on Guido, not GCP.
Are they porting to the AppEngine Python 3 APIs? Hell no: those are totally useless (no BigTable, no user authentication, etc.). If you have to rewrite everything to replace all the functionality that used to be "batteries included", you might as well build something can run on any old bare VM instead, and so—like any sensible external developer—my colleagues are doing everything they can to avoid depending on any Google-specific APIs, libraries or platforms.
I really though AppEngine and PaaS were the future, but evidently that future will not be written at Google.
That's hilarious(ly bad): BigTable was like the main selling point of OG AppEngine, and they practically forced their user authentication scheme on you too.
Which in practice meant that "migrating to a new database under a different account" translated into "rewriting half a codebase due to a cascade of breaking changes in the surrounding tooling".
> > I know I haven’t gone into a lot of specific details about GCP’s deprecations. I can tell you that virtually everything I’ve used, from networking (legacy to VPC) to storage (Cloud SQL v1 to v2) to Firebase (now Firestore with a totally different API) to App Engine (don’t even get me started) to Cloud Endpoints to… I dunno, everything, has forced me to rewrite it all after at most 2–3 years, and they never automate it for you, and often there is no documented migration path at all. It’s just crickets.
GCP will fail, I think everyone knows it, and even if they don't, they fear it.
Though our engg. team is on it, Pointless hours of productivity lost :-(
it actually winds up being less DevOps work, on average, to support open-source systems running on bare VMs, than to try to keep up with Google’s deprecation treadmill.
For context, I tend to get dumped with undocumented poorly written untested DS code bases every time I move to a new job.
The difference between setups where there's a bare VM/server and it runs using standard tools (command line scripts) or at least some open-source orchestration and any kind of proprietary technology tied to infrastructure is massive.
Additionally, the server stuff can be maintained and debugged by a far larger number of people, thus reducing bus factor.
Like, maybe in the web dev world there are things that everyone knows that are AWS services (maybe S3 counts, I guess) but in the DS world, I just know that I'm in for a whole lot more pain if they haven't just used bare servers.
Keeping your stuff up and running, patched with the latest security updates takes time. And opensource systems also break compatibility at times. Sometimes the installation/upgrade process changes over time and you need to learn how to do proper upgrades even if the software technically can be made to work with the rest of the system.
It's hard to quantify, but I don't think it's so clear-cut as Yegge makes it look.
sure FOSS needs to be upgraded too, and sometimes there is change that you need to adapt to, but you are in control of the schedule, and if you are busy you can delay the upgrade work to when you have time for it. a service being shut down won't give you that flexibility
Google gives you 12 months notice before turning off the lights.
A FOSS tool "forces" you to upgrade when the old version you're using has a serious security issue that's is not backported. Which means that unless the supported version is fully backward compatible with your software, you have a quite tight schedule to deal with.
Sure, you always have the "option" of running insecure systems. Unfortunately a lot of people choose that "option" and severely underestimate its cost...
an upgrade with some incompatible changes is still a world of difference compared to potentially being forced to rewrite a whole bunch of code because the system you used is completely discontinued and the alternative that replaces it isn't even remotely similar
I know they deprecated SimpleDB but I don't think they ever actually shut it down for existing customers, that might have changed in the last few years though.
1) If you're busy maintaining backward compat, you're not busy building innovative new things -- Microsoft. 'nuf said.
2) You have a built-in excuse to not make your new stuff equal in functionality or greater to the old stuff, because hey, customers can just use the old stuff right? I built some software that talks to O365 Sharepoint. Ok, I have two choices of API: the older Sharepoint API or the newer Graph API. They recommend Graph, ok. Build the product and put it into production. Get a new customer requirement for fine-grained auth, and then find out that Graph doesn't handle fine-grained auth -- only the Sharepoint API does that. Oops. Ask Microsoft when that capability is coming to Graph? No timeline, because the Sharepoint API isn't deprecated.
3) Keeping old stuff working may be easier, but building new stuff is harder because the landscape is muddier -- how many times have you looked at Microsoft docs and found multiple ways to do things with no idea if the docs you're looking at actually apply to the approach you're using now?
I think there are ways to have (mostly) the best of both worlds. Linux does it by bring as much stuff in-tree as they can -- they break compat all the time for out-of-tree stuff. Ever try to keep a proprietary VMWare module building without errors? In-tree KVM never breaks because they're super careful with the ABI. Some languages have explicit mechanisms by which they attempt to keep things current, but make it easier on users -- Kotlin is an example of this [1] but its a young language so time will tell whether they can actually thread this needle. My experience so far is yes, I think they can -- I've updated code from Kotlin 1.2 to 1.3 to 1.4 with relatively little pain.
[1] https://kotlinlang.org/docs/reference/evolution/kotlin-evolu...
"But that's sooooo muuuuch too teeessstt!! Whaaaaaaaaaaaaaaaa!!!!!!!!!!"
But that's your job dude. So what if it costs more money and time. Fucking pay for it anyway.
I feel like startup software dev needs a swift kick in the ass or three with that phrase.... "do it anyway!"
Who fucking cares if these founder assholes only make an 800% return instead of 900%? Do it right.
It's Google who doesn't provide backwards compatibility, upgrade paths and migration options.
> But that's your job dude. So what if it costs more money and time. Fucking pay for it anyway.
So why doesn't Google pay for it anyway?
> I feel like startup software dev needs a swift kick in the ass or three with that phrase.... "do it anyway!"
You mean Google devs.
I've been on the web since the mid 90s. The last 15 years have had more cool things taken away from me because of sunset bullshit than the first half of that time period. Software (even web apps) should be eternal, there should never be "features removed" or "EOL" announcements. If they don't want to run something they should open source it and let the community take over. What the hell is the point of hoarding all this wonderful IP if it serves noone?
The focus on consistent behavior and once launch, never deprecate practice has caused many headaches, I personally know of control planes specifically written to accommodate old behaviors that a few customers had years ago even when the entire service has been updated. But IMO this is still much better than deprecating production services and behaviors on the fly.
This isn’t true, of course. They supported managed Kafka before AWS did: https://cloud.google.com/confluent
If you have an issue it’s not GCP support or GCP’s eng job to fix the cluster. There’s real service uptime implications of this.
- Datastore -> Firestore in Datastore Mode: This was a seamless transition with no API change, Google changed the underlying tech.
- Firebase -> Firestore: Never happened. They are different products. The Firebase Realtime Database still exists.
- gcloud CLI updates: It is a CLI tool, it gets new versions.
- App Engine with Python 2.7: It still exists, you can still use it. There is a new App Engine v2 that supports Python 3.
- Cloud SQL v1 to v2: They had a tool to automatically upgrade, it set up replication and switched over.
AWS has new versions too. RDS Aurora v1 (MySQL 5.6) can't be upgraded in-place to RDS Aurora v2 (MySQL 5.7)[1]. AWS Lambda runtimes get deprecated[2].
[1] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...
[2] https://docs.aws.amazon.com/lambda/latest/dg/runtime-support...
I migrated all my "working set" stuff to Python 3 piecemeal over a couple of weeks and never looked back (except when I needed to pick up some older project and take an hour or so to refactor it).
The "There Will Come Soft Rains" reference, however, was brilliant. I hadn't thought of that story for _decades_ and it hit home.
I work in visual effects/animation, and 2020 is literally the first year anyone is even trying to use Python 3 (and Python 2.x) is still supported everywhere. Prior to 2020 Python 3 couldn't even be used at all (and I mean that literally: could not be used outside of isolated toy examples on alpha software that wasn't actually running in production).
We (as an industry) have written thousands and thousands and thousands of business-necessary logic/scripts/libraries in Python 2.x that have to be updated to Python 3 that work across commercial tools so literally every tool has to be updated at the same time in tandem to Python 3, so much so that the VFX Platform[0] project was created in 2014 specifically to address how badly Python 3 had fucked everyone in my industry.
It's all make-work too: literally none of us actually need Python 3's new features.
Javascript's "use strict"; is the way to do backwards compatibility if you really really really need to change the semantics and keep existing code working as-is.
Is there much Flutter and React Native dev going on at big companies outside of Google and Facebook?
I know plenty of smaller companies are using them because they don't have what sometimes feels like near-infinite money to throw at mobile engineers, but I can't think of a single Big Name On Blind with iOS or Android position listings that mention Flutter or RN. They're all native.
Even at Google and Facebook, usage of either of their respective frameworks seems to be an exception and not the norm, and they're still just components in a larger native project. I wonder if the goal is to eventually re-write everything? But that seems impossible given the size of these applications' source.
Taking two of the examples of "bad platform moves": Python 3 and Apple.
Where I work, we are simultaneously dealing with Python 3 and GCP deprecations[1], and it sucks. But Python is more popular than ever![2] Yes, some people (including us, at least for our website code) left Python, but lots and lots of people have moved to Python 3.
I've seen lots of people say "oh, the move from 2 to 3 wasn't that bad because of the tooling." That's probably true, and we probably wouldn't be moving to Go were it not for the fact that we had API changes _and_ Python's changes to deal with (and Go gives us other benefits).
And then there's Apple. Apple has a long history of maintaining compatibility for a time, and then getting rid of the bits that are holding them back. There are tradeoffs.
Maybe Android doesn't break things for developers, but Apple makes it so that iPhone users can keep updating their phones to the latest OS (and apps) for much longer than the typical Android phone. Because of deprecations and a general desire to make apps that fit the platform, developers of popular apps on iOS put the time into updating them, and these apps are really nice to use as a result.
I think fit and finish on non-first-party native Mac apps tends to be better than those on average on Windows, though that's just my opinion.
Carrying backwards compatibility comes with tradeoffs, and it's possible to be successful by making different tradeoffs than maximizing backward compatibility.
[1]: https://blog.khanacademy.org/go-services-one-goliath-project... [2]: https://redmonk.com/sogrady/2020/07/27/language-rankings-6-2...
> So when it comes to shouldering the burden of compatibility, you need to pay for it. Not us.
Surely if there are shoppers/customers affected by a deprecated service on GCP (or any cloud provider), then that would indicate a paying customer behind it. That payment should at least cover costs of running that service (infra, development, software and maintenance). In that case, why deprecate at all? - version the API and continue running it as long as it economically makes sense?
> In the Emacs world [...], when they make an API obsolete, they are basically saying: “You really shouldn’t use this approach, because even though it works, it suffers from various deficiencies which we enumerate here. But in the end it’s your call.”
> Whereas in the Google world, deprecation means: “We are breaking our commitments to you.” [...] It means they are going to force you to do some work, possibly a large amount of rework, on a regular basis
It's fine if they want to go hunting for their Perfect API v10™, but just leave the old APIs alone - build abstractions under them if needed. Don't push that work onto your customers.
First: Why don't more programs or services go the `make-obsolete` route and allow people to use deprecated APIs forever? If maintaining compatibility is the goal then I can't see how removing a function in a new API version will ever work. Maybe it's because companies are expecting people to be dumb and call support when they use this deprecated API that was specifically designated as unsupported. "But I can use it," they'd say, and still complain that it breaks or something. Maybe that workload is what they're trying to avoid. But maybe they'd silently move away instead if continuing to use the platform meant rewriting everything.
Second: Why don't people usually have automated API refactoring tools? When I was trying to port a Minecraft mod to a later version I found a bunch of things had changed in the engine. Okay, go to MCP and look at what changed. Guess what? It's all a bunch of Markdown. They painstakingly went through the process of labeling every single API change and method renaming and so on, and they put it all in Markdown.
There was so much missed potential there. Imagine if you had a policy encouraging your developers to put down every API change you make in your program in a machine-readable, schematized format that could be printed to a webpage or Markdown or something else, but could also be combined with a static analysis tool to at least print out the parts that need changing, if not refactor them, just like Google's internal tooling? If developers don't notate those changes by hand then that information is going to be permanently confined to unparseable changelogs and obscure IRC conversations. That's the kind of thing that a semver increment can never capture the nuance of. Why guess what broke or manually parse changelogs based on a major version increment when you can have a computer do all the work for you?
Maybe it's because a changelog is "information to throw away," that you'd have the painful time trying to read through the moment things break and then can forget about a couple of days later when the refactoring is done. But the changelogs and migration steps are essentially a bridge closing a chasm, with your entire userbase on one end trying to get to the other, and if it's not made easy enough for them to cross they will either give up and leave or stay on the other side forever.
Anyone understand the technical jargon here? I'm a bit lost.
> Bigtable maintains data in lexicographic order by row key. The row range for a table is dynamically partitioned. Each row range is called a tablet, which is the unit of dis- tribution and load balancing.
And later...
>The master is responsible for assigning tablets to tablet servers, detecting the addition and expiration of tablet servers, balancing tablet-server load, and garbage col- lection of files in GFS. In addition, it handles schema changes such as table and column family creations. Each tablet server manages a set of tablets (typically we have somewhere between ten to a thousand tablets per tablet server). The tablet server handles read and write requests to the tablets that it has loaded, and also splits tablets that have grown too large.
> I considered using Google Cloud Bigtable for my online game, but it costs an estimated $16,000/year for an empty Bigtable on GCP. I’m not saying they’re gouging you
Ok, but they are
As someone who hasn't been burned by it personally, it is hard to quantify the actual risk and maintenance cost of GCP's deprecation policy, but I know decision makers at larger organizations that are rightfully afraid of it.
Aws used to be simpler , while they are screwing it up now, the recent route53 upgrade was terrible , a single click action has now become 4-5 clicks they are still better than GCP.
Azure has the really the best interface (outside of DO) , it is easy to go to related and nested resource in one freaking app. Both google and aws end up forcing you to open three four tabs to setup interconnected apps .
DO interface is really really good ,however they don’t have the complexity of features that the big three do , so not a fair comparison
Also I haven't worked _that_ much with firebase, but it seems like a great example of the benefit of using GCP. Firebase is a cohesive and accessible solution to a lot of what can be fairly nightmarish technical problems. This kind of thing will always depend on the project/team/team size, but I'm just trying to say that there are significant benefits to GCP that should be considered.
I can get Microsoft on a call anytime, I have account managers responsible who talk to me atleast once a month, reach their product teams, get preview access, get the MS account manager of my customer to help with a deal, put them in front of my customer, help with compliance, even get their sales guys to recommend my product. To a lesser extent I can do a lot of that with AWS too, with even sub $100k/year spends they will still put an account manager for you. I am not sure any of this was possible with GCP for most customers.
It is extremely hard to get a human from Google to talk to you even at $100k+/year GCP spends. Google has contracted a lot of partners to do all the heavy lifting in support for them so they don't have to do the hard work. It does not work, the partners can not do much beyond what is available on the portal or clarify beyond the documentation.
Azure serves enterprises really well, and has really made effort with developers and their support for startups in fantastic, AWS is not as good yet, however it feels like they are really trying and their tech popularity works in their favour and they care about backward compatibility a lot S3 API from 2006 still works. With Google and GCP it does not even look as though they are trying at all.
An interesting note is that Amazon actually planned to EOL part of this vis-a-vis what path to address objects at, and then walked it back (for buckets created before a certain date): https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-... I think the behavior is still considered "old style" if not explicitly "deprecated", but is "supported".
This is exactly the kind of commitment I expect from Microsoft or Amazon, it is why enterprises pay premium for a product.
In a similar situation I imagine Google would have just sent a note and with a window of few months and shutdown the old API. Great for innovation and keeping your tech cutting edge, not so much for the customers who are not as agile as they and can only move slowly if at all.
[1] https://www.backblaze.com/blog/design-thinking-b2-apis-the-h...
Why is that guy still giving money to GCP if it's so bad?
Go back to AWS. There's a reason if AWS is still the market leader.
A phd quit my employer and went to google. On last week we were discussing relative differences. He pointed out searching for X really had no wrong results and even if you didn't like it who you gonna call? Google probably is a little siloed from customers in a way many other business are not. Our IT systems deal in money. There's nowhere to run and hide from customers.
I've used AWS and Azure gig.current uses GCP (don't get me started on that) but what is the fourth?
The only thing that’s weird about this is that it’s about 3 years late.
If this had been written 3 years ago it would have been topical.
But really it’s old news that google deprecates everything and leaves it’s developers in the lurch and that as a result no one wants to use their stuff.
It’s not surprising to hear this from Steve Yegge cause he’s a super switched on guy, but it’s surprising to hear this from him NOW.
Google’s cloud ship sailed years ago, they just haven’t yet got to shutting the whole thing down, but the writing is on the wall.
Google seems totally oblivious to how much developers distrust them, which is probably because dump trucks turn up every hour or so and dump cash onto everything which probably makes everyone feel that everything they do is awesome, even if it isn’t.
Just search google for the topic of google cloud shutdown, many speculate it.
https://www.theinformation.com/articles/google-brass-set-202...
I really enjoyed the essay and surely is some truth, but these products have all been pretty good for us.
The issue is that developer confidence is very low due to google being in a never ending relentless drive to cancel stuff.
Of course, those might count as one product, in which case, it's 13 years. I'm not sure how Google's algorithms work. Does it detect suspicious behavior per-service or per-cloud-account?
Ironically, so few people use Google in production that successful production use IS anomalous behavior.
For example, emacs lisp suffers from bitrot. Some core things haven’t changed in forever but also you can’t pick up elisp config from 15 years ago and it’s fine today.
All live systems evolve and change. In this case emacs is just so slow that Steve thinks that it doesn’t.
But, as you said, he is a good story teller. Like Taleb. Getting the moral across is more important, even if sometimes the wolf eats little red riding hood in malformed renderings.
By comparison, I have nothing. zilch, for my hard work. Neither do most people working on core tech at AWS. That's why they empathize with their customers (I'm guessing).
Fun fact for you - if I get promoted this year at Amazon to L5, I'll _still_ make less than a new grad at Google. It's extremely depressing to the extent that I can't get out of bed in the morning, but I'd wager that inferiority complex probably helps on the product side to empathize with customers.
This doesn't appear to be close to true based on levels.fyi, unless you're severely underpaid for an Amazon L5.
*The net effect is "pay for performance" isn't really a thing.
The only way for me to do a significant compensation bump is to do what's called a "dive and save", but this is very rarely done for L5's and is significantly more common for L6's. I got an offer for about $200k a year ago at a hedge fund, but that wasn't eligible for dive and save because of my level and I'm (clearly) too mentally defective to pass a Facebook or Google loop.
The other more popular approach is to "play the game" as I think of it, I.e. every 18-24 months you switch to a new job with a substantial increase in pay as result.
I wish this wasn't the case, but I guess it is what it is.
"Just move" is something someone without any empathy for other people's circumstances would say - most people aren't capable of "just moving" as if the only concern is choosing an employer.
No disagreement on the good recommendation as a signal, but I doubt that'll matter to the HC if I can't pass 3/5 rounds at the minimum.
That long enough part is the kicker and that terrifies me in every job search (to the extent I have sometimes jumped at the first half decent opportunity, and agreed to salaries below my potential). I find it does help to apply often and everywhere and not be invested in any particular opportunity.
With dating they say every failed relationship is a step toward finding the right one, and I think that same principle applies to job interviews.
Anyways, that’s my two cents
Mind you, he would probably have turned us down anyway to go do a PhD, so he probably dodged a bullet.
You make six figures at a top tech company and you’re bitterly complaining because you think you’re entitled to more, and in the same breath you’re chastising Google’s pretentious culture? There’s a lot of irony there.
Focus on improving yourself and don’t get caught up in comparing yourself to others.
I see too many people talking about their Tahoe Google offsites (jbd@ on twitter claims a bunch of her coworkers just work from Tahoe during ski season) or their Colorado skiing weekends.
Anyways, hypocrisy doesn't mean I'm wrong, and I think its valid to be mad that I'm seen as inferior daily.
Yet you still make more than what the vast majority of engineers in western Europe will ever make in their whole career. And that's still only looking at the most developed countries.
The job market isn't fair. No use getting burnt over that.
I'm also quite surprised by your wage comment: a common aphorism is that Amazon cheaps out on everything except real estate and compensation.
Which ones do and don't? Because all of the ones I've heard of have pretty loose ones compared to my organization.
And yes, Amazon doesn't cheap out on real estate because we own so much of it. It's honestly remarkable how it's done. But I was under the impression everyone knows they pay less.
I only have about ~40 stocks up from about 35 before the refresh cycle (unvested) and that's unlikely to go up if I get promoted (if you have too many they don't give you more). Googler and FB refresher values are significantly higher.
It's not unique to Google. Lots of organizations intentionally build a culture of elitism. However, it means I would never, ever rely on Google for anything. Free products like search are a-okay. My email is on gmail because legacy.
Build a business on Google platforms? Nope.
I don’t even work on the “core tech”, I am on the consulting side, probably make less than you do (albeit in a much lower cost of living area) and I am probably older. But, I am not throwing a pity party on HN
I’m not even here to shame $150K. That’s what I was making as CRUD developer pre-Covid at 45 years old (long story) and I also graduated from a no name state school - in the 90s. $150k still means you’re making more than roughly around 80-85% of American households.
I fully understand shaming $150k, new grads at Google make $180-$210k.
You graduated from college and are now making more than most developers in the US and you work at a big tech company. You have the opportunity to leverage that and either get promoted from within or change jobs.
Are you really struggling making $150K straight out of college? You graduated from a state school (as did I) you probably don’t have that much debt.
For context, I’ve been out of college almost a quarter century and I am just now making a little more than a college grad at Google. I’m not complaining and I only have a good 10-15 years to take advantage of the opportunity. You have your whole life ahead of you.
Stop complaining, get on the grind, and do what you need to do. No one owes you a quarter million just because you walked across the stage with a CS degree.
I think the real problem is I'm not capable of doing so, at least according to the fair labor market.
> Are you really struggling making $150K straight out of college? You graduated from a state school (as did I) you probably don’t have that much debt.
I don't have any debt, I just have a really pathetic savings rate and a really low net worth compared to nearly everyone I know, including folks that didn't do "the grind" that I did. I'll probably not be able to retire or own a car, much less own a house someday.
I hate that we as a society can say that if you have signs of cancer go see an oncologist and it’s not okay to say that if you show signs of mental health issues go see a professional.
But everything about your comments and your user name points to it.
I don't like the IQ/talent fetish that seems to pervade Anglophone culture. Some people have a stronger natural predisposition to sit and stare at a screen and figure out puzzles, and that happens to be sought after on the labor market. Doesn't mean others have less inherent value as human beings.
Would be nice to have a small appartment one day. I just try to offer some perspective.
Dude, please seek mental help. You have so many working years ahead of you. Stop worrying about owning a house. Start budgeting using something like YNAB and get your savings rate under control. Also get all the basic investment vehicles going: 401k, after tax Roth, backdoor Roth etc. You’ll do just fine.
A lot of my friends are already putting money down on houses and are making hundreds of thousands in options trading on principal that I _don't have_. I'm falling behind permanently and even maxing out my 401k (which I didn't do last year because I don't anticipate living to 65 but decided to this year) won't get me closer with a low principal.
Not living in that area anymore, I can't say. I haven't lived there for 10 years.
I think the Pixel division is the only exception, though I don’t own one so I can’t comment personally.
As far as I'm informed every Pixel version so far had battery degradation issues. After year or so the battery seems to have only a fraction of the previous capacity.
I am reminded of this funny Feynman quote (paraphrasing): "The whole purpose of the existence of the Mensa club was to decide who else was eligible to join the club". Now, in all fairness, this is true of all the big tech companies to varying degrees, but boy does Google Cloud in particular take the cake on this one!
Of course, Big Tech does produce very good products, and at scale, but that is being built by about 10% of their engineers. The rest of them work there just to keep the interview loop as exclusive as possible. :-) After all, otherwise, one of them would have actually solved long runningcustomer problems.
Plus the rest of the world is now coming to an agreement that a lot of Big Tech's growth is coming from a lot of unfair advantages - such as massive data collection before it was understood as a bad thing, ability to lobby local politicians at a much bigger scale, ability to influence regulations that hurt them somewhat but entirely kill the competition (GDPR for example), complete lack of accountability for past malpractices (e.g. Facebook's friendly fraud case, for which no one went to jail) because they are now too powerful to actually be sent to jail.
I feel like this was super true in the pre Facebook days but they were humbled A LOT by the massive flop of Google plus.
I honestly don't feel like I get the my s * doesn't stink vibe from them anymore.
Especially because everyone knows their Achilles heel now.
But for their ad revenue what would they be?
Everything else material from came from an acquisition or is infra stuff.
It's cool but they also had to learn the hard lesson of Hadoop too.
Yeah they built big table... But the rest of the industry standardized on Hadoop (at the time) because they never released an implementation and so they started losing out on good hires because they didn't want to get locked into proprietary Google infra.
I think you see them recognizing this with intiatives like all of the stuff around kubernetes.
But hey I wouldn't build anything on their (non open) stuff and get more scared by the day over my dependence on El Goog because they've shown time an again how little they care for us common plebs.
But I feel like they get better when they lose but they've only started losing more recently
(Rant over)
Hadoop and Bigtable are not the same thing. Bigtable is a noSQL database. Hadoop is a big data processing framework. Hadoop is actually an open source implementation of MapReduce, which was developed at Google and which has since been replaced by Flume
Hbase was open-source implementation of Bigtable while HDFS was a copy of Google File System. The divergence appeared with other processing frameworks like Spark but now there's Apache Beam to unify it all.
I do believe leadership is failing Google, and their lack of vision is starting to affect the company. There was quite a bit of inertia, but I fear search might drop in the next 5 years and there will be nothing ready to replace ads.
I had a phone conversation with a hiring manager once, and obviously I was not the right type or class or caste or something, but it wasn't a technical screen at all, so it's a mystery to me forever what exactly determined the "in group" as you put it.
As such, I know I'm biased towards them, but I can't really assess the overall company.
You talk as if people from FANG (your invented acronym to exclude Amazon) look down on all others, but it seems that you are the one looking down on all others, everyone who wasn’t lucky enough to get into G/FB straight out of college while also building up not just 1, but 2 or 300k or net worth.
These companies you aspire to do behavioral interviews, and I’m guessing you’re failing them.
Take a step back. You won’t “fail” life because you “only” got a job at one FAANG and not another, or because your net worth is only 100k straight out of college....even the sentence I just wrote sounds completely ridiculous.
Your problems are stemming from how you view yourself and the world, which is full of false assumptions. A lot of smart people on HN have told you this —- you should listen to them.
Don't know why people say this. I'm usually pretty good at behavioral interviews. The problem is the algorithmic interviews (again, this is where my IQ fixation comes from).
IDK man, compared to most everyone else I seem to have failed life at this point given that I have zero accomplishments and little financial security.
It sounds to me like you have anxiety. I'm saying this because I have anxiety, so I know what it's like. One hallmark of anxiety disorders is that you think things are true about the world, even though you don't actually have evidence this is the case. Or, you fixate on some pieces of information while ignoring evidence to the contrary, which again is irrational thinking. Or you hold assumptions about how the world works that aren't proven. You should look into cognitive behavioral therapy as a potential solution.
You say "compared to everyone else". But, you have to acknowledge that the net worth and income numbers you named put you into the 99th percentile, right? So, you actually mean "compared to a very small subset of people I have failed", right? And this is just a completely irrational argument. I mean according to this vein of thinking, then since I don't have as much money as Jeff Bezos, I've failed. And the implication then is the whole world has failed. Right?
> "compared to a very small subset of people I have failed"
Compared to nearly everyone at elite universities I have absolutely failed. I'm sure some of them will become the Jeff Bezos of 2050 as well - it's irrelevant that it's still a small number. I don't want to be compared against somebody that works at Cisco or IBM for instance for the work and mental anguish I've put in.
You also have an implicit assumption that people who work at Cisco or IBM haven’t put in work or mental anguish, and you’re wrong. I know many people who worked and studied hard to get jobs at these places.
You’re artificially restricting the set of successful outcomes so that you can say you’ve failed because you don’t fall into that arbitrary set. Not to mention that your entire definition of success as defined by things like money and perks is wrong.
[1] https://www.google.com/amp/s/www.cnbc.com/amp/2018/09/14/sta...
Please realize how offensive and insulting this is.
You were hired by one of the most successful tech companies, which most people could never aspire to. You claim that Amazon pays you 80% of what (you assume) Facebook or Google would pay you, which puts you financially ahead of the vast majority of people. By calling yourself a failure, you’re calling nearly everyone on Earth an even bigger failure, which is offensive and makes you look extremely entitled and detached from reality.
> Don’t know why people say this.
It’s because you come off in your posts as entitled, obsessive, elitist, insensitive, and bitter. These are the only sideS of yourself that you convey here, so that’s why people assume you’re failing your behavioral interviews.
Sorry again if I’m being harsh, but you’ve been posting the same stuff here for months, and every discussion gets derailed by people consoling you or advising you, which you invariably reject. Let’s stop going in circles.
Let me repeat: that is thousands of millions of people who can barely even conceptualize the life you're able to live.
On another note, I’m in my mid 40’s spent all of my time bouncing between yet another CRUD job most of my career (not complaining, it’s paid for a decent lifestyle in my relatively low cost of living area) and I’m always encouraging fresh CS grads to go for broke and take the r/cscareerquestions route of “grind leetCode and work for a FAANG”, knowing they will make more fresh out of college than I did until a few months ago.
Someone else’s success is not my failure.
However, as a University lecturer I know a lot of people Google have employed. In my opinion, they have rejected some of the greatest students I have ever taught, and accepted some idiots who know how to speak well.
While they have employed some good people, I believe they purposefully taret the type of people who think working at Google makes you a fundamentally better person than anyone else, rather than the best programmers/researchers/AI/whatever.
There's no question that Google is happy to have false-negatives in interviews.
> I believe they purposefully taret the type of people who think working at Google makes you a fundamentally better person than anyone else
why would they do this? And, if they do this, how are they generally speaking so successful? (this is also an amusing comment because the only group I find more critical of Google than HN is Googlers)
The usual recipe for success in tech: competitors who are even less competent.
See also: how Microsoft dominated personal computing back in the day.
Again I think this whole line of thinking is nonsense, but even if you take it at face value, it still doesn't make sense.
By making illegal deals with PC sellers, and by vandalizing apps running on their OS.
They have almost complete monopoly on online search, web ads, online video, on phone OS, on browsers. And they are not afraid to abuse those to get more, and are getting away with this.
You don't need to do anything right when you rent-seek most of the online world.
'Rent seeking' doesn't refer to the same kind of 'rent', of course, but I think the analogy holds up pretty well. All Google has to do is turn a few screws, and the rest of us will have no choice but to sing whatever tune they call.
When a company has a lot of money coming in, they can be successful despite decision X, rather than because of decision X.
Microsoft can interview people asking about filling airliners with golf balls. Valve can just not bother with Half-Life 3. Google can have zero support and a self-driving car division that keeps avoiding chances to release anything.
That the companies are successful doesn't mean all their actions are smart - sometimes it's that their successes are big enough the occasional bad decision doesn't hurt them.
Also, even if I am right, real-world competence may involve more bluffing, less high quality knowledge of algorithms, than I would like.
I’ve met plenty of “smart people” who couldn’t get things done to save their lives partially because they couldn’t communicate well or play well with others.
This has been a source of angst for friends of mine at google for over 15 years, which is almost 3/4 of google’s existence. The hiring process is just random.
Part of it is due to measures put in to avoid certain unconscious biases (hire your friends, regardless of how good they are; hire only people like yourself, etc). So I have some sympathy.
But only some.
This was prompted by me noticing that most of the people who still had their badges visible after work appeared to have Google badges.
- sales
- finance (explains itself really)
- SRE (mostly because of favourable time zones)
Most of the core engineering is in London/Zurich. Dunno what's gonna happen post Brexit, but anecdotally, the FAANG which I am most familiar with has had real problems hiring in London post-2016.
I was originally planning to go to go for the native k8s experience.
Then I read somewhere online that was doesn't sunset services where as gcp does.
I was half way through the migration process to gcp and bailed on a dime.
Microsoft still wins on the desktop. The only way to beat them was to compete in new emerging markets.