HNHacker News
TopNewBestAskShowJobs

trunnell

1,290 karma · joined August 9, 2009

2009-2019 at Netflix.
submissionscomments
trunnell··on Being “rockstars”: when software was a talents/creatives industry
I think the "rock star" analogy did more damage to our profession than most care to admit. Rock stars get treated a certain way, they are catered to, their egos might get fluffed up, etc. None of those behaviors are compatible with working in software teams, and none of the "best" developers I know want to be treated that way, anyhow.

That said, I think this article is essentially correct. I would put it like this: building information machines out of pure information is a creative endeavor. Human performance tends to fall on a bell curve, and IMO it's true for developer output in particular. So if you want to create a hugely valuable solution to a big problem, you should probably hire the best people you can find and organize them in a way that lets them max out their creative power.

The main obstacle to this IMO is poor leadership attitudes & practices. This article's author said it best in another post:

  "Boring" is a good strategy if you think the bigger existential risk for your company is that Product & Engineering will [...] fail to ship a reliable product on a reliable schedule. But if you're more afraid of the business risks of shipping average product, at an average cadence, over the course of years, you should consider deviating, at least sometimes, from the playbook whose entire definition is "optimized for safety."
https://morepablo.com/2022/04/against-boring.html
trunnell··on Software Development Ethics
Look past the flaws of this particular code of ethics.

The better question: Is it time for us to adopt some code of ethics?

Many other professions have important ethical standards. Medicine, law, civil engineering, and many other fields. These standards are broadly respected by the public.

Surely by this point we can all see that software is often just as critical as bridges, medicine, justice, and other public goods. Should we not hold ourselves to some standard?

trunnell··on Reliability: It’s not great

  > Flyctl for AWS
Have you tried AWS Copilot? I’m having good success with it. Probably not quite as simple as flyctl, but still it’s only one command to deploy a container.

I would really like fly.io to overcome these hurdles. I bet they will.

trunnell··on FCC commissioner says government should ban TikTok
You're right, this is probably the main problem (rather than the ability to change the content on TikTok.)

So you're implying that their future influence operations could be enhanced and better targeted by using profile-level viewing history on TikTok, similar to how Cambridge Analytica used FB profile data to target political advertising?

trunnell··on FCC commissioner says government should ban TikTok
Right - it's complicated and nuanced. Some Disney censorship is probably benign, right? This is the argument made by the crowd urging inaction.

Consider the recent example of John Cena, an actor in the "Fast and Furious" franchise, apologizing to China for calling Taiwan a country. Is that apology good, bad, or irrelevant? Probably irrelevant by itself, I'd say.

But what about China's ability to coerce John Cena into making that apology? This is more akin to the problem you might see on TikTok. Influential people that others look up to parroting Chinese propaganda, wittingly or unwittingly.

It doesn't require much imagination to see the problem. If China secretly coerced 10 famous actors to say, "the US should leave Taiwan to China," and those videos had >100 million views in the US, that could be a very successful influence operation. TikTok could be a means of distribution for that kind of thing.

trunnell··on FCC commissioner says government should ban TikTok
People underestimate the power of TikTok to influence opinion.

Strangely, TikTok makes up a large part of the "information diet" for a lot of people. It's tough to find a great analog from 1930, but I think the postal service analogy is in the ballpark.

trunnell··on FCC commissioner says government should ban TikTok
The executive branch is responsible for foreign policy and national security.

This issue is about a hostile foreign government being in a position to manipulate the internal communications and political debate in the US. So it's more in the realm of foreign policy and national security rather than US corporate law.

(Surprising to see this comment downvoted - why?)

trunnell··on FCC commissioner says government should ban TikTok
Duh.

No reasonable adult (in any country) would go on the record to propose that a hostile foreign government should run any part of their internal communication infrastructure. It's well documented that all companies in China are beholden to the political whims of the Chinese Communist Party; when national security is at issue, Chinese companies and the Chinese government are one entity, sadly.

I'm squinting skeptically at all the comments here that are confused about this, especially the ones with extra helpings of carefully chosen rhetoric (whatabout-ism, free speech, free markets, etc). There are no "free speech" issues here, ByteDance is a foreign company, not a US person. This is a national security issue.

If you think it's cool to allow TikTok in Western countries, please explain why it would have been fine to allow the Nazis to run the Royal Mail, the US Postal Service, and one third of the broadcasting services in the 1930s. Would that have been a "free speech" or "free market" issue?

The Chinese government has a long way to go to earn the trust of other countries.

See this list of common Chinese propaganda themes to check if they may have already influenced you. https://en.wikipedia.org/wiki/Propaganda_in_China#Common_CCP...

trunnell··on Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf]
That's the essence of adaptive streaming - the player continuously selects a video bitrate that can be downloaded quickly enough given the network conditions. That requires multiple video bitrates being available, and each one must be encoded such that the player can switch between them at known cut points.
trunnell··on A list of developer questions to ask prospective employers
A few of these are good, but others remind me of that "How To Professionally Say..." list that was making the rounds a few months ago, which was actually a list of snarky, passive-aggressive and shockingly unprofessional phrases.

By all means, ask your interviewer good questions! But please don't say this:

  "Does your code review process promote empathy?"  

  "Tabs or spaces?" 

  "Does the company provide snacks and/or drinks?" 
Ugh.

When I was a junior engineer, I thought that the tools I used and the products I worked on mattered most. But now, after two decades, I can see that what actually mattered was the people I worked with. If you're interviewing, focus on the people you meet. Are they kind to you? If you had to correct a mistake of theirs, how do you think they would react? How do they interact with each other? Would you enjoy spending time with them? Do you feel like you might learn something from them?

Take these questions (from the article) and focus on the interpersonal parts:

* How do you estimate work?

* How often does your team interact with other teams?

* If we have a very successful year, what would that look like?

Some questions I would add:

* Can you tell me about someone who was promoted recently? Why were they promoted?

* Can you tell me about some feedback you gave someone? How did it help them?

trunnell··on Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf]
The OpenConnect team at Netflix is truly amazing and lots of fun to work with. My team at Netflix partnered closely with them for many years.

Incidentally, I saw some of their job posts yesterday. If you think this presentation was cool, and you want to work with some competent yet humble colleagues, check these out:

CDN Site Reliability Engineer https://jobs.netflix.com/jobs/223403454

Senior Software Engineer - Low Latency Transport Design https://jobs.netflix.com/jobs/196504134

The client side team is hiring, too! (This is my old team.) Again, it's full of amazing people, fascinating problems, and huge impact:

Senior Software Engineer, Streaming Algorithms https://jobs.netflix.com/jobs/224538050

That last job post has a link to another very deep-dive tech talk showing the client side perspective.

trunnell··on Electric vehicles are way more energy-efficient than internal combustion
Eh, it's more like... exactly what the title says: EVs are just more efficient.

1 gallon of gas contains 33.7 kWh of energy. [1]

The average ICE car goes 24 miles on 1 gallon of gas. That's 0.7 miles per kWh (24/33.7 ≅ 0.7).

The Tesla Model 3 goes about 4.1 miles per kWh. That's 5.8x further per kWh, or 5.8x more efficient!

Most of the difference is because ICE engines are inefficient. They waste energy. Just like the article says. :)

The point made by the article about the source of electricity from power plants was simply that EVs are more efficient even when charged from coal fired power plants! Your point about the "ICE in some power plant somewhere" is temporarily true... as the grid de-carbonizes there will be fewer and fewer emissions from grid-scale generation. (Nuclear + renewables is my personal guess for where we're headed.)

It's kind of sad that several threads here are playing devil's advocate. Unless some dramatically better tech takes the world by storm in the next 2-3 years, EVs are going to completely take over the market. Economies of scale should dramatically drive down costs over the next 5-10 years, and then all new cars will be EVs. No one will want the old, expensive, dirty, loud ICE cars. And this is really good news for everybody!

  [1] https://www3.epa.gov/otaq/gvg/learn-more-fuels.htm
  [2] https://afdc.energy.gov/data/10310
  [3] https://ecocostsavings.com/electric-car-kwh-per-mile-list/
trunnell··on Netflix lays off about 150 employees
Former Netflix manager here. Netflix frequently competes for L6/L7 talent against Google, Apple, and FB -- and wins. Look at the detail view on levels.fyi and sort by compensation. They pay extremely well for the best talent.

But that's not really the point. Your job safety is ultimately about whether other companies would hire you. And if you're good enough to get a role at Netflix, then you can get another job extremely quickly here in the valley. Most top recruiters know that Netflix is very selective. I have yet to see anyone who was let go at Netflix struggle to get another job. They usually get multiple offers, in fact.

trunnell··on Netflix lays off about 150 employees
Netflix's keeper test doesn't apply to someone going through a personal problem or other temporary issue. A manager would support them, making sure they get the time off they need. (And Netflix managers are usually very generous and understanding in that regard.) The keeper test is about maintaining a team of great contributors.

But the "dream team" doesn't last forever. The metaphor is meant to reinforce that. If you could play for a season with Michael Jordan, Barkley, Ewing, and the other amazing players from the actual 1992 Dream Team -- wouldn't you do it? And if you got cut the next season because there was someone even better than you -- would you regret being on that team? It would probably be a great experience, even if it only lasted a while. This is what the "dream team" idea is about.

It's clearly not for everybody, not for every company, and not for every situation. Netflix's culture is all about excellence, and the "dream team" and "keeper test" ideas work well there. I saw it play out first hand.

trunnell··on Netflix lays off about 150 employees
This is exactly how the opening reads to the book No Rules Rules. Erin Meyer, a business professor, has that exact analysis of what "should" happen in an environment like one Netflix describes. But she interviewed dozens of Netflix employees for the book and found that the theory you state doesn't hold. The book is about figuring out why that is the case.
trunnell··on Netflix lays off about 150 employees
I can't make sense of your comment.

> you don't pay more to person A to entice person B to enhance your team

That has nothing to do with the "dream team" idea. It's a metaphor referring to the 1992 USA Olympics basketball team [1] that included:

  Magic Johnson, 
  Michael Jordan, 
  Scottie Pippen, 
  Patrick Ewing, 
  David Robinson, 
  Charles Barkley
... and a few other of the most famous all-star basketball players. The coach was the famous Mike Krzyzewski of Duke. This was made possible by a rule change that allowed professional basketball players to play for the first time in the Olympics.

This team went to win the gold medal. The closest score in the eight olympic matches was 117-85. In other words, the "dream team" utterly dominated.

Netflix does not try to make do with what they have. They try to assemble a "dream team" for each thing they do. I saw Netflix be pretty successful at it while I was there.

[1] https://en.wikipedia.org/wiki/1992_United_States_men%27s_Oly...

trunnell··on Netflix lays off about 150 employees
No, not at all.

I was there for 10 years and I don't recall any mass layoffs. The article says this is about reducing expenses. Note that the article doesn't say if the layoffs were in Los Gatos or in the Hollywood office, or both.

The keeper test is mostly about helping managers be effective when they need to remove someone from their team. It eliminates the bureaucratic red tape that most companies have for firing someone. Managers at Netflix are still expected to give feedback to everyone about their performance and if there is a problem, give someone an opportunity to improve. No one should be surprised if they are let go for performance reasons.

The difference is that at Netflix someone who isn't keeping up will not be subjected to a 1+ year long process driven by HR before they are eventually fired. Instead they will be given specific time by their manager and if the problem isn't fixed fairly quickly, the manager can decide to let them go.

Most people have been on a team where there was somebody who was constantly slowing everything down or causing other problems. If you're a manager, you know that those people end up taking a huge portion of your time. At Netflix, managers are effective at removing those people.

trunnell··on Actual is going open-source
If 200K sounds high to you, check out the salaries posted on Levels FYI [1]

The Bay Area might have the world's highest paying market for software engineers.

[1] https://www.levels.fyi/?compare=Google,Facebook,Netflix&trac...

trunnell··on Pay Transparency: European Commission proposes measures
If pay transparency is the goal it might be better to publish each employee’s pay. That way, at least it’s possible to connect the reasons why someone is being paid more or less (like experience, or extra responsibilities, etc). Few people are excited about that, though.

I’ve set pay for engineers at several companies. In my experience, employee pay levels have a very broad range with the max and min overlapping significantly between levels, making the levels not very useful for pay analysis. This is because of 2 things:

1) “Equal work” is mostly a myth in creative jobs like software. It’s an idea that we’ve inherited from factory work where the job types were highly regimented. In software and other creative fields, if you ask two different people to solve the same problem you’re likely to get two very different solutions. Across hundreds of people, the best solution might be 10x better than the average. That’s a big difference, and not like factory work at all. If someone is doing 5x-10x better work, and your company can make use of the surplus, should that person be more highly payed? What if your competitor is willing to pay more than you are currently paying her? Would you match? And what if matching would push up the top of the pay band?

2) Usually, the goal is to match the market pay (although this is sometimes laundered through various levels of abstractions like years of experience, degrees, etc). Levels can rarely be made detailed enough to model the full complexity of the market. Let’s say you have two mid-level engineers with 5 years of experience and similar work histories. One builds web UI and another is a security engineer. These two people are in very different markets. Security expertise is hard to hire right now (and has been for the last 10 years) and therefore pays 1.3x - 2x. But your company only has one level for “mid-level engineer.” You have the option of underpaying the security engineer, and thus expecting to lose them to another company willing to pay market rate, or pay them market rate yourself but have pay levels with enormous ranges.

trunnell··on We sound like idiots when we talk about technical debt
A "culture of approvals" is one root cause of this situation. Something needs to be done, but instead of doing it an engineer asks someone else for approval and doesn't receive it. Healthy teams don't work that way.

In a healthy team, engineers are responsible for the health of the system.

In a healthy team, engineers make decisions about non-product-feature related work, including tech debt. Product managers decide the relative priority of product changes. Engineering managers (who must be just as technical as the engineers on their team) make schedule decisions, balancing the health of the code base and the product needs.

trunnell··on We sound like idiots when we talk about technical debt
Sorry to tell you this, but it sounds like you have a bad manager. Your story is like not how healthy teams operate, which in my experience actually are much more like the post you're replying to.
trunnell··on The unlimited storage that Google promised my university is being discontinued
Why do you say it'd be a bad idea to store frequently accessed experimental data in Google Drive?
trunnell··on Ask HN: Where do you find your sense of purpose working in the private sector?
Think of profit as a natural way to make an activity self-sustaining. And if a product is profitable, it likely has an authentic reason for continuing, namely that it’s creating value for customers. Now, of course, judgment is still required to ensure your profitable activity is good for society, ethical for everyone involved, etc. But to me, I’d rather be working on something that meets the criteria above than a non-profit or gov project that requires donations (or gov appropriations), because those things can continue (or be halted) for very unnatural reasons and there’s no guarantee the value being created exceeds the effort put in. Hope that helps you!
trunnell··on Amazon more than doubles max base pay to $350k
When I've recruited people somewhere within their multi-year vesting schedule, I've found that they absolutely hate the "golden handcuffs." It leads them to resent their employer and ultimately motivates them to leave.

Delayed vesting is a terrible practice. Perhaps we'll see more companies offer immediate vesting of equity comp as a way to compete for talent. This is what Netflix does.

trunnell··on Amazon more than doubles max base pay to $350k
> this move seems reminiscent of Netflix's pay structure, in the sense of offering a higher cash portion but lower equity.

Netflix gives you 100% of your total comp in whatever form you choose: all cash, all stock, or any combination in between. Also, Netflix's stock plan vests immediately-- they don't try to lock you in with 4-year vesting. They view stock compensation as something you've earned, so there are no strings attached.

It's well known that for top roles, Amazon's pay ranges are often well below their peer companies. You can see this on https://levels.fyi. So it makes sense that Amazon is trying to catch up.

trunnell··on Don't start with microservices – monoliths are your friend
Ok, maybe a better way to say it is that having teams independently develop services is a good way to reduce the coordination tax if you have high coordination costs. If your environment doesn't have that problem I guess this doesn't apply.

Coordination between engineers was a frequent activity everywhere I've been regardless of how well built the systems were. For example: a new requirement for the customers signing up in a given country to have features X, Y, and Z enabled. In a large organization there are probably a few teams that will be involved that make that happen. The question is how to coordinate them.

Many companies try to solve it with top-down decision making, prioritizing certainty but hampering productivity (some teams have to wait) and strictly limiting risky innovation (nothing can be done without approval).

Independent teams (each developing independent services and acting without top-down approval) is a different way to coordinate development that values productivity (keeping everyone unblocked) and innovation (finding better ways of doing things).

> You have to invest a bit in tooling and recommended patterns to keep things from going crazy, but you kind of need to do that either way.

Aha, here's a difference. If we're talking about the same things, common tools and patterns don't need to be enforced. Independent teams can pursue very different patterns without needing to agree with each other. This is a big advantage if you don't like being told what to do by people who are pretty remote to the problem you're solving. Different people react differently to that. Netflix teams tended to be staffed with very experienced and skilled people (no junior or mid level positions) so there wasn't much value in one engineer dictating architecture, patterns, or tooling to another. Nearly all tooling was opt-in, and the good tools were the de facto standards. But if you came up with a better tool or pattern, you had the freedom to try it out. This is how independence fostered innovation, and why microservices were useful in such an environment.

trunnell··on Don't start with microservices – monoliths are your friend
I assume you're referring to the visualizations like the ones halfway down this post? https://netflixtechblog.com/vizceral-open-source-acc0c32113f...

It wasn't really a horror, and those charts are a little misleading. I'll try to explain.

Plenty of others who were also there at the time might see it differently, but IMO this was a trade off to get higher productivity and higher resiliency at the cost of higher complexity. So it was a feature not a bug.

When the cloud migration began in 2010, we had a straightforward architecture of a handful of Java monoliths and a giant Oracle database all running in a local data center. The DVD service made all the revenue but the future was the streaming service. It would turn out over the next 10 years we would need to grow engineering headcount over 10x to support the growing business complexity. Some of that complexity was foreseen and the intent was to address it with a culture analogous to a cellular ecosystem.

There are many successful designs in nature that evolved to a complexity beyond our current understanding, like the human body or even the protein signalling pathways within individual cells. These systems tend to have excellent homeostasis and survivability in the face of novel stimuli. Notably, each cell is fairly independent: it can receive signals but it acts on its own, and its internal complexity can be hidden from its neighbors.

We created a culture where teams were independent units. The sayings were, "loosely coupled and tightly aligned," and, "no red tape to ship." This is the opposite of an architect making top-down decisions. Instead we prioritized local decision making within teams which tends to be fast and effective, so this worked out well for productivity. But the side effect is that the overall system grew a bit beyond the limit of any one person to easily understand it. Thus the automated tooling to visualize the graph. But the dependency graph itself was rarely a problem. Any given developer was usually able to trace all the requests they were responsible for all the way down the dependency graph, and that's what really matters for development and debugging. Generally, no one needed to grok the zoomed out picture -- even during outages, problems were typically root caused to a particular service, and not related to the whole dependency graph.

But the dependency graph makes for a compelling graphic, so it gets people's attention. The real story is, "productive organizational culture made of independent units."

trunnell··on Can We Regulate Social Media Without Breaking the First Amendment?
I'm curious about the difference between free speech and the right to a megaphone. From what I've gathered, the problem isn't that social media publishes harmful posts at all, but rather that social media promotes bad posts to tens of millions of people. Isn't that a difference that regulation could rest upon?
trunnell··on Illuminating the Dark Forest
> A path out of the dark forest, or at least towards finding a guiding light that could illuminate it, lies, in my opinion, in improving the current provenance system through algorithmically-aided positive reinforcement and better documentation.

Do you mean that the underlying protocols should provide the standard tools of observability and know-your-customer practices in order to enable the existing regulatory system to do its job?

trunnell··on Illuminating the Dark Forest
> The problem in today's space is that WE CANNOT KNOW! The infrastructure for documenting and fighting scams simply isn't there. We're not capable of resolving disputes...

This is an obvious problem to anyone familiar with traditional banking and finance. I feel bad for the cryptocurrency true believers who didn't see that coming. Even if you take as a given the most idealistic vision for permissionless money and smart contracts, the lack of legal oversight is ultimately a fatal flaw. Unfortunately it's a fundamental part of these ideas.

← PreviousPage 3 of 8Next →