744 karma · joined May 16, 2017
A risk that has a 1% chance of wiping out 100% of your company, vs a risk that a 10% of chance of costing you 10% of your company, are not the same even though the net "expected value" is nominally 1% for both.
Google noticed this, and for unique/low-traffic search terms, was able to synthetically generate enough "fake" traffic that the "fake" traffic became the dominating signal for those terms, and therefore Bing started directing users to the fake results.
The bad thing here IMO is the level of tracking of users via this toolbar, but fundamentally this seems as bad as any other digital fingerprinting or advertiser tracking as anything else that's become common on the web. This is not to excuse Microsoft for doing a bad thing, but it really had very little to do with "scraping Google", which somehow became the popular media takeaway for this.
EDIT: a decent contemporaneous article in Wired: https://www.wired.com/2011/02/bing-copies-google/, mentioning how Microsoft was using the clickstream data from the browser/toolbar, not scraping Google results per se
EDIT: ofc in the Twitter thread, there's few disputing that if this is actually true, or if maybe Apple does different things in different markets, etc. so everything with a grain of salt I suppose
TLDR: it's pretty debatable
Few observations for the OP:
- These are potentially signs of depression in general, that nothing seems worthwhile and you don't want to engage with any activities. Are things outside of work still engaging? Do you put your energy into other hobbies? Or does everything feel tinged with this same flavor of generalized discontent and oh-whats-the-point?
- Setting aside depression as a factor (which OP should seek diagnosis + treatment for, if relevant), in general this is also a self-fulfilling prophecy. By believing that your work has no meaning, that just being barely-better than the rest of the slow/lazy team is good enough, OP will make this into their reality. So in a sense, it's not an objective observation about FANG work-life, and more of a subjective decision to shape OP's reality into this mode.
- OP could be doing better work as you suggest (notice I said better, not more), improving their tech skills, adding impactful examples to their portfolio/resume, empowering their future career, but they're choosing not to. It's not as if the manager has actually blocked OP from learning or working on interesting things. And even if the team is a strategically unimportant team with no long-term career prospects, OP would still be hurting their career by not actually using this time more effectively, to add to their own portfolio of work or body of knowledge... So what if they won't get recognized or promoted for it on their current team, OP will still need to have something to talk about in future interviews or future promotion cycles on other teams. If someone asks OP, "what did you work on from 2019-2020, what significant value did you deliver, what did you learn, what skills did you gain?", what's OP's answer going to be?
- Speaking of promotions... it may feel to the OP that "coasting" along is going okay for now (i.e. no complaints from the manager), but I doubt they are at a career stage that they can really "coast" infinitely/indefinitely. In 2-3 years (or maybe sooner), someone will ask, "hey what has this person really done, that's meaningful or impactful? how critical are they to the team? feels like they're just coasting, right?". Even if OP isn't PIP'ed, they would still be extremely unlikely to earn any promotions, simply because, what is there to justify it? What next-level work have they done, if all they are doing is coasting? So, this is a fleeting/temporary situation, that OP is fooling themselves into thinking coasting like this is sustainable (because short-term, it is).
IMO, the US will almost certainly make up a lot of ground on the deaths per capita metric (or leapfrog entirely), given the explosion of new cases starting in June and the lagging growth in deaths that comes after cases. Scroll down to the "Deaths" section with the weekly per capita "heat spectrum" of sorts, and the US recently has reached red hot over the past few weeks, while European counterparts have been much cooler: https://www.nytimes.com/interactive/2020/world/coronavirus-m...
With containment, everything else is possible (healthcare system not overloaded, economy can be opened up, schools can reopen, etc.).
The US has capitulated completely, and even containment isn't really on the table nationally; on a state-by-state basis, some states may achieve a level of containment, but others are simply not even trying (or like Georgia, actively banning containment measures like local mask enforcement orders).
- If you are working a company that has broken incentive structures, the first and immediate question you should answer for yourself is, what do you want to achieve there, as a manager? Are you really going to try and fix these deep cultural issues, and try to improve the company as a whole (or even just your department), and its whole internal mechanism for decision-making (which is broken and based on bad incentives)?
- If so, great! Commit to that problem, build support, get alignment, and who knows you might actually succeed, and end up in a C-level role for your efforts of turning the ship around. It's unlikely, but if that's your goal, then have at it.
- If not, then who cares? Accept the risks of what you signed up for, get whatever it is done that you need to do, and leave when the time is right. Understand your place in the hierarchy, understand what your personal goals are and how you can achieve them, and do whatever you need to do, without worrying about the health of the company or how other decisions are made. If you can't change it, or the effort to change it is out of the scope of what you're personally trying to achieve in your career goals, then it is literally a waste of time and energy.
---
On the topic of indicators or lack thereof, this is literally what good managers do. They find insights that others miss, and are successful in either making winning arguments or in framing debates differently and changing the success criteria. These powers can be used for "good" or "evil" to be blunt, that's up to the individual manager. You can practice at a small scale, for example for operational improvements by creating new metrics/KPIs for your team to better monitor service health in different ways than they used to. You can apply this to business decisions as well, by providing new analyses of how out-sourcing did not actually pay off, and defining new metrics for how to make those decisions (given that you've uncovered some hidden factors that were missed in the initial evaluation). There's no magic spell to it, it's just work and experience and critical thinking, which you gain through repeated application of curiosity and tenacity to problems that others tend to overlook or are too stuck in their ways to think differently about.
Personally, I loathe projects where I have little autonomy, and so I proactively seek out problems that other people don't want to own or have avoided owning or simply didn't understand or appreciate their important, but which I believe to be important problems (which depends on the prerequisite that, I actually must have some real insights which are unique that others have missed), and so I go out of my way to carve out ownership of those problems. Because nobody understood them before or wanted it before anyways, I don't get much pushback, and I can use these opportunities to make decisions autonomously, and also build up trust and support by demonstrating success with such projects. Eventually, I can leverage my banked trust store and positive balance of success, to push for changes or influence decisions in other areas that are more contentious and may not have gone "my way", had I not done the work to build up trust previously
- A key part of the financial crisis is that, collectively the group of auditors and risk evaluators and experts who are supposed to understand/evaluate/quantify risk, generally failed to properly price in the risk of mortgage-backed securities failing en masse (they basically said, the chance of this happening is astronomically low, and were wrong by many orders of magnitude). Taleb's point was that, for certain events, it is surprisingly difficult to accurately assess their true probability, and even very advanced, highly-paid experts (with a strong vested interest in getting it right), can fail (and fail spectacularly). One of the lessons here is, to not place too much mis-placed confidence in certain kinds of estimates, and to be skeptical of gambling on certain probabilities which are fundamentally non-verifiable or hard to verify (even though they might look like regular probabilities and come packaged with lots of nice documentation and fancy equations and backed by lots of PhDs who tell you how wonderful their models are... we still need to be skeptical). When the weatherman tells you there's a 40% chance of rain, you can be sure that even though he might be wrong, he is very unlikely to be wrong by orders of magnitude (i.e. that the actual chance of rain is 4% or 0.004% or 100%); when a quant at an investment bank tells you that a AAA-rated MBS will fail with probability 0.00001%, you probably shouldn't trust that number in the same way, even though both are probabilities and both come with models and equations to support them.
- With COVID-19 and pandemics in general, it's not clear that the relevant public health experts failed to price in the risk or to accurately estimate/approximate the risk. What seems more likely is that our governments and businesses failed to heed their advice (i.e. that pandemics are inevitable, that they will happen with some regularity e.g. every 10-30 years, that in a globally connected world they will spread more rapidly than ever, etc.), and failed to act on the advice they were given (or in the case of the US, almost literally threw out the pandemic response handbook and ignored it). This makes it less of a "black swan" problem, and more of a regular (though still spectacular) failure of governance.
I might argue however that, there is perhaps a general tendency to incorrectly categorize such things as "black swan" events, which is an interesting phenomenon in and of its own (i.e. why do people want to take white swans and paint them black or to simply pretend as if they were black when they are not).
- "I'm a rich/powerful/successful media person. I had success in my industry doing business a certain way, so why shouldn't I be able to translate that into running a tech startup too?"
- "I'm a good programmer and I don't see any reason why Zuckerberg would be smarter than me. What's so special about Facebook? We can beat them."
- "I'm in touch with what most people want, and this product idea totally makes sense to me. All I need to do is polish it up, and I'm sure I'll find users who are as excited about it as I am."
Or, the flip side of the same coin is really about ego. One of the hardest lessons of the lean/agile/tech-y start-up culture (in its idealized form), is to not let ego interfere with learning, but also to be able to let go of ego-centrism without giving up confidence or competence (i.e. you don't have all the answers and you should be fundamentally humble and curious, BUT you should not forget or overlook your existing skills or competencies either).
It is much more profitable to charge recurring "subscription" payments for treatments to keep you healthy-ish, rather than to give you a one-time cure that actually fixes you or removes the disease entirely. This already is a thing: https://www.cnbc.com/2018/04/11/goldman-asks-is-curing-patie...
There is simply no way to align healthcare with capitalism, without creating perverse incentives around every corner.
It's also kind of a difference in strategic approach, to pool resources and create cutting edge institutions, or to adopt a more decentralized approach where yes, you might not get as quick of a pace of innovation, but each community is more self-sufficient. So, you might not ever get a fancy cure for cancer or CRISPR technology in this alternate universe, but most communities might have their own local clinics and more local nurses and maybe better overall health outcomes by focusing on common treatable problems, rather than pushing for cutting edge innovation. Similarly, you might have fewer PhD's, but more high school graduates or bachelor's level of education.
Anyways, nothing about this is all that realistic or anything, just some idle world-building in my head.
Basically, with 3D printers and solar power and local grids and machine shops to fabricate for local needs, and a loose mesh network to connect these decentralized nodes/communities together (like Mastodon, instead of the paper-and-print Amish newspapers that exist). A kind of techno-libertarian-Amish blend, to form a society that was more resilient and modular, rather than hyper-centralized, dense, and easily swayed by viral influence/behavior/trends or literal biological viruses.
EDIT: Of course it would never "scale" since such a system probably could not support higher population levels at all, but maybe that's okay in the ultra-long-run or some post-apocalyptic rebuilding future (also dangerously veering into population control and genocide-ish topics).
So, even without a mandate, the market forces at play in terms of proposing roadmap projects and choosing what to invest in, will naturally tend to favor those solutions with can pull double-duty (solve an immediate problem/pain-point that Amazon already has, and with a clear path/potential to be converted into or evolve into an external sell-able service too).
Essentially, an employee of the facial recognition provider forwarded an "investigative lead" for the match they generated (which does have a score associated with it on the provider's side, but it's not clear if the score is clearly communicated to detectives as well), and the detectives then put the photo of this man into a "6 pack" photo line-up, from which a store employee then identified that man as being the suspect.
Everyone involved will probably point fingers at each other, because the provider for example put large heading on their communication that, "this is not probable cause for an arrest, this is only an investigative lead, etc.", while the detectives will say well we got a hit from a line-up, blame the witness, and the witness would probably say well the detectives showed me a line-up and he seemed like the right guy (or maybe as is often the case with line-ups, the detectives can exert a huge amount of bias/influence over witnesses).
EDIT: Just to be clear, none of this is to say that the process worked well or that I condone this. I think the data, the technology, the processes, and the level of understanding on the side of the police are all insufficient, and I do not support how this played out, but I think it is easy enough to provide at least some pseudo-justification at each step along the way.
One random example from 2018 (so, probably a lot more advanced and widespread by now): https://www.adweek.com/digital/adobes-newest-labs-project-ca...
In a sense, you're like an athlete who doesn't take steroids or doesn't push the limits of doping policy by trying to game the countermeasures, and instead of feeling aggrieved that others are achieving similar/better results as you via their just-within-the-letter-of-the-law tactics (or sometimes wildly outside the law tactics, a la Varsity Blues), instead you still feel some allegiance to an idealized view of a fair system that simply doesn't exist.
Compare that to the opaqueness of police misconduct/brutality internal investigations, or how frequently even the bad actors that are known to have done wrong, are still granted their pensions. Where is the equivalent payout for a restaurant owner who doesn't follow health rules? Where do you see other restaurant owners banding together to defend a restaurateur who's had their misconduct exposed? Where do you see something equivalent to "qualified immunity" for the food industry?
In other words, this is a great example of a reasonably well-functioning market. If regulating the police behaved much more like regulating the food industry, that would actually be fantastic. In many ways, that is exactly the goal of the protests, to bring a similar level of transparency and accountability and high standards to policing, as most people already expect and has long been standard for the food industry (in developed countries).
30 years ago, if you worked in software, then you were almost certainly building shrink-wrapped products, that had to "go gold" before being printed onto 1000s of CDs at a factory. How quickly/easily could you change anything? Not for years, typically. Hence, you were very careful to try and build things right the first time, but the ironic thing is, with such long feedback cycles (i.e. you never really got feedback), that all this waterfall-style effort to plan everything and do everything right the first, rarely ever materialized as perfect/seamless UX for the end user anyways. It turns out, you need to iterate to improve.
Nowadays, the pendulum may have swung too far in the opposite direction... it could legitimately be said that, some things are too easy, tinkering with changes is too easy, and thus nobody expects to have to maintain something for a long period of time. If the cost of rewriting everything is low enough, people will be tempted to just do that whenever they change their minds about something, rather than forcing themselves to predict the future upfront and make durable decisions that will survive for 5-10 years out in the wild with no changes.
IMO, there's no single right answer for all scenarios, all products, all forms of engineering, etc. But it's explainable why such trends may emerge over time.
RDS SLA's are here (doesn't mention backups though, so not sure how that's handled): https://aws.amazon.com/rds/sla/
Not a lawyer or anything, but my layman's understanding is that, you essentially are voluntarily opting-in to waiving liability when you sign up for AWS and accept the terms and conditions, and instead of liability, you agree to accept service credits if SLAs are not met.
Semi-/un-related anecdote, but looking back at what my team was like 6 years ago, we used to be a classic enterprise data warehousing team, and our worldview at the time was basically that, "everything is just a bunch of databases or file systems, and our job is to copy (literally scp) files from source A to destination B." All problems (literally, all) would be solved as one of, "how can I export a file from this source, and what kind of table do we need to create in the DWH?" By that logic, we would make decisions like, "why do you need Splunk/Promethus/Datadog/etc. for operational monitoring, when you could just write those metrics to tables in the DWH?".
In retrospect, what is surprising about such a worldview, is not that it was so limiting or that it hurt us in the long run (it did, but less than one might expect), but how far you could actually go and how much you could actually achieve, with such simple primitives.
One milestone for success I think is, when your feature repository can be used to regularly gleam new insights that are surprising (even to your experts). So for example, imagine you are a typical retailer selling widgets, and you have 10000 customers who bought widget X, and you want to identify out of your remaining 10 million customers, which of those customers should you target as likely to want to also buy X? Now, of course, this is a classic problem and doesn't require a fancy feature repository; just the basic order information of who-bought-what is enough to create the classic collaborative filtering solution from.
But with a rich/useful feature repository, you can throw this problem in there, looking for the most similar other customers compared to the 10000 you know, and to rank which features are most powerful (in terms of, why it makes them similar), and you should see some interesting insights that you never would've guessed on your own.
Just to answer the "what really sucked" question a little bit, even if it's implementation-specific... man it really sucked to start out with 2-out-of-5 level of knowledge about Apache Spark, and to stumble through learning both the obvious lessons and the non-obvious/esoteric bugs/quirks, until after 1-2 years the team finally felt comfortable with like an 4-4.5/5 level of knowledge about Spark. This was also specific to the state of Spark back in ~2017 or so, and it's come a long way since then.
I can see RDBMS being reasonable for small-to-medium size of data, but beyond a certain threshold, I think it starts to breakdown (at the few 10s/100s of TB level, maybe?).
- Years and years ago, we already had a data warehouse (DWH)
- In the data warehouse, you would store data like, each and every order that all customers have made (i.e. up to and including full-fledged facts and dimensions about each)
- Now, let's say axiomatically/hypothetically, a very useful and highly predictive feature for ML is "# of orders made in past 7 days" for each customer
- Can this be computed from the data already in the DWH? Yes, absolutely, but it's a new computation and not an existing column/attribute in the dimensional model.
- What if you need to recompute this feature daily, for millions of customers and orders? Well, we could always just add it to the dimensional model, compute it once, and let people just use it/share it... but why? Most internal users of the DWH probably don't care about something like "# of orders past 7 days" as something to be added to a customer dimension or per-customer grain (too specific or whatever), and moreover, the DS/ML folks want the same feature but for "every 1/3/7/30/90/180/365/730/etc." days breakdown, as well as a bunch of variations about orders and things other than orders (e.g. "average time between new orders, over past 7/30/90 days" or "average $ spent over past 7/30/90 days", as features that serves as a proxy for frequency of activity and level of engagement)
- Hence, it makes sense to keep the "golden copy" of data in a canonical form in the classic/standard DWH as a baseline, and to separately/independently compute features out of that data and to store them in a different system (which can also be optimized for the different query/access patterns that DS/ML have, vs traditional BI). Over time, it also made sense in certain cases to go upstream of the DWH to source data from and process it more directly (for performance/efficiency reasons), though generally deriving features out of the transformed dimensional models was still very useful.
- It took our teams ~1-2 years to really go through this evolution and reach a mature-ish state, but for the past 2-3 years, we've benefited tremendously from having an independent feature repository/store, that is separate from the classic DWH. Benefits came in all the obvious and some non-obvious ways, i.e. in faster iteration/cycle time, in better quality/repeatability, and in being able to automatically discover interesting relationships that no human could have anticipated - simply by having a very broad/large repository of features and running automatic feature selection over it.
Does the value that a worker provide the company actually go down or not, if transitioning to working remotely? I don't think it's so clear-cut as to be an automatic and obvious no (that it doesn't go down). If management thinks it does, then they will only be proven right/wrong over time through the labor market and attribution or through financial results.
Of course, the opposite could well be true, and this how a market opportunity is created: managers who think the opposite, can put their money where their beliefs are, and hire/pay workers based on a different value-rubric, and if they're more correct in their assessment, then they might win out in the long-run.
Personally, I don't think deeper integration with traditional fiat currency/banking systems is a sign of failure; rather, it's the first step towards this possible future (which of course, is highly unlikely still, but 20 years ago it was 0% likely, 10 years ago it was 0.00000000001 likely, and now, who knows maybe 0.0001 likely?).
Of course, this too, is all speculation. It's hard to predict the future.
Some reasons why (loosely paraphrasing internal guidance on how to run meetings and why writing culture matters):
- PowerPoints are lazy/evil, and not good formats for conveying complex ideas; by nature, PPT's encourage shallow thinking, and is thus antithetical to the goal of having rigorous analysis/deep-thinking in making complex tech/business decisions
- Docs should stand on their own, without needing to be presented or walked-through by a person; forcing you to explain your ideas and support them in such a way that any sufficiently motivated reader can follow along and work out what they mean on their own, is again a way to encourage you to be rigorous and thorough (but limiting you to 6-pager format, for brevity)
- Writing long-form paragraphs and full sentences forces you to take time to refine your ideas
- Forcing people to take time to digest the doc first, gives everyone a chance to take notes and formulate ideas/responses to the info/arguments presented; this saves a lot of time as compared to, walking through a doc/slides for the first time, and having a roomful of people encounter those ideas for the first time and to ask questions in real time as they encounter them... reading the doc should hopefully already address the most common questions/critiques, and so you can spend more time on 2nd/3rd-order more advanced decisions or arguments
Early settlers are incentivized to grab large swathes of land quickly and cheaply, to get the system bootstrapped. Later settlers have to fight over smaller tracts of land, because after all the land is finite (you eventually reach the ocean). However, the land is also nicely divisible into smaller and smaller sub-plots, so units can be adjusted as needed.
The rules by which this "westward expansion" are governed are written into the Bitcoin protocol. But unlike land and the ocean which are natural facts for the most part that we take for granted a priori, with Bitcoin the "border"/limit here is also defined as a theoretical construct (and hence, could theoretically be changed, but this would be a hard fork and people would have to reevaluate the value of a new network with a new set of boundaries/rules).