Brendan at Intel.com
brendangregg.com
brendangregg.com
* https://www.youtube.com/watch?v=tDacjrSCeq4
* https://www.brendangregg.com/blog/2008-12-31/unusual-disk-la...
* http://dtrace.org/blogs/bmc/2008/11/10/fishworks-now-it-can-...
† A play on Lockheed Martin's "Skunkworks".
I didn’t expect this to be so… literal.
Reminds me of Bret Victor's Inventing on Principle (2012), https://www.youtube-nocookie.com/embed/PUv66718DII
I have to sit down for a minute...
My experience with this in interview loops is that it's less about admiring a persons technical abilities, but more a checkbox question to determine whether a person fits into a certain role model that companies have set up. At most FAANGs, interviews will expect that you mention you are being the tech lead of a team (5-10 engineers) at a (L|E)6 role, even if one isn't a manager. At (L|E)7, it will be 50+ engineers. As a regular engineer, one would probably have an issue getting hired at a high seniority level without answering the question the right way. Things might be different for well-known personalities like Brendan.
The TL is the person who the buck stops with on technical discussions. While part of the job is providing mentoring for junior engineers, they aren't directly responsible for performance management, headcount allocation, etc, in the same way that a people manager is, and people don't usually directly report to them in the org chart.
By definition, no large engineering effort is solitary. It makes sense for the more senior eng to be responsible for the decisions that will affect more people, and that requires talking to and understanding them. If people want the senior eng title, they need to be able to do that kind of work.
In some cases you want your three wizards supervising the architecture of 100 people working on 25 products… and sometimes you want them lovingly crafting every line of a framework or other core component that is going to mechanically influence all that work even more than design review ever could.
I want roles in companies for the technical artists to progress. Where you get harder problems, work on more complex issues, do things that require more research and autonomy to solve. These skills are also valuable and needed in companies and most don't have a way of retaining that talent as they have no track for them.
And some problems require not large numbers of people thinking a little about it, but rather long periods of time uninterrupted for one or a few people to think deeply about.
In my own career, I am currently pursuing independent consulting as the "progression" of an individual contributor role. I believe this is fairly common, and I think there's kind of a mutual cause-and-effect going on there with the lack of good senior IC paths at many companies.
Most companies don't have any issue bringing in the aging expert that will code and lead their internal team, teaching them the in-outs of the technology they should master, yet have serious issues having such skillsets in-house.
Reason for that is simple - in most areas you can achieve more, if you can properly guide dozens of engineers than lock yourself in the basement. Main exceptions are crazy specialized perf gurus.
I think this is a problem in all companies in general.
Walk in the door to find that move of the L7s I've encountered have been leading 50 person teams (as manager or TL). Seems to range from more like 20-35 and, among ICs, there often not actually the only TL, just the most senior on the team.
On interviews it sucks. The engineers assume I’m some jackass manager guy, and the manager guy thinks I’m in the weeds.
It's unfortunate how https://github.com/janestreet/magic-trace and PMUs in general can't be used by lots of people using cloud VMs.
- PEBS (Precise/Processor event based sampling, so that we can accurately get instruction pointers on PMC events)
- uncore PMCs (in a safe manner)
- LBR (last branch record, to aid stack walking)
- BTS (branch trace store, " ")
- Processor trace (for cycle traces)
Congrats on the job.
I admit that I don't know how Intel works, but disabling the use of these performance-counters at startup should be sufficient for any potential security problem.
I'd expect that only development boxes (maybe staging?) would be interested in performance counters anyway. Maybe the occasional development box could be setup for performance-sampling and collecting these counters, but not all production boxes need to be run with performance-counters on.
When I profiled the D compiler with and without PGO enabled it became obvious that a lot of the speedup of PGO basically comes just from running the program, the choice of testcases made almost no difference.
I also really want them in CI but that might be a long way away.
Production is exactly the place where you want full performance counter support, all the time, everywhere, on every machine.
1. FaaS
2. Containers
3. Lightweight VMs (e.g., Firecracker)
4. Bare-metal instances
The ship has sailed on neighbor detection BTW. There's so many ways to know you're a VM with neighbors that disabling PMCs for that reason alone doesn't make sense.
In the crudest sense of "do I have a neighbour", sure. Of course, that's hardly secret -- if you're in EC2 you can just count your CPUs to figure that out.
But there's more questions you can ask:
1. Is my neighbour busy right now?
2. Is my neighbour a busy web server, a busy database, or a busy application server?
3. Is my neighbour hosting Brendan's website?
4. Is my neighbour hosting Brendan's website and he's logged in writing a blog post in vi right now?
5. What's Brendan writing right now?
It's not immediately clear which of these questions can be answered using certain capabilities! Few people would have guessed that you could read text off someone's screen using hyperthreading prior to 2005, for example. (Pretty simple although I don't know if anyone has published exploit code for it: Just look at which cache lines are fetched fetching glyphs to render to the screen.)
1. Meta: https://twitter.com/brendangregg/status/1515482126871044098
Caught me a bit by surprise as well, as Intel seems to have stagnated a bit as of late, but the opening paragraph seems to indicate Brendan thinks otherwise, and who am I to disagree.
I wish you luck on the new adventures, and hope you'll have tons of fun!
Their core CPU design has managed to stay relevant (and profitable) despite issues with production, and has regained some measure of success with their 12th generation.
Their production business has a good roadmap, but success in execution remains to be proven.
GPU design is a bit of a wildcard, and I'm excited to see if it pans out.
I would say it's fair to say that what came out of Intel certainly felt stagnant as the past decade drug on, but there's still promise that they remain competitive.
That has varied. At one point, you could set a "please don't sabotage performance" environment variable that made e.g. Matlab 2x faster. Then it started being ignored. Making software deliberately cripple performance to make your hardware look better reveals the kind of company it is.
A lot of “we can’t do that” or “we don’t do that” in big corps actually comes down to “we don’t think we need to do that.” Because why mess with success?
But when everyone realizes success is slipping, a lot more things become possible. I’m going through that now at my employer and big ideas that bounced off of walls for years are now getting done in months. It’s fun.
The best place for an engineer to be (in my observation) - is at a company that faces a fierce competition. To envision, design and build a pipe pumping gold vs to exploit such a pipe when it is built - requires different skills.
>I recently worked with another hardware vendor, who were initially friendly and supportive but after evaluations of their technology went poorly became bullying and misleading.
I wonder who this is.
https://mobile.twitter.com/GregL_Intel/status/15207891560240...
This is something I am so excited to hear about Intel. As a consumer and user, I want Intel/Apple/AMD/NVidia all to be competitive and pushing the boundaries of what is possible in computing.
So far Apple has done an amazing job with M1. AMD has had super success with Ryzen. Nvidia has had great success with GPUs.
Recently, it seemed Intel was lagging, in large part due to their culture. It is exciting to see them get some "geeks" back in charge.
It is an exciting time in computing!
Qualcomm is designing desktop-level ARM chips to compete with Apple's M series. The team came from Apple via a startup acquisition (Nuvia). Will be interesting to see how that turns out.
One project that I would find really interesting would be to leverage the insane capabilities of GPUs and 3d graphics to make extremely detailed visualizations of millions and billions of performance events, intervals, code paths, data structures, heat maps, and the like. It'd be great to get views beyond the raw data that are not oversimplified and useless, but rather to get the feeling of being at the helm of some seriously detailed (and precise!) data with zoomable resolution and a lot of assistance of analyzers to surface visual artifacts. I think it'd be both entertaining and highly productive to engage our visual cortex more.
The blog post title is "Brendan@Intel.com." HN formatting ruined a little bit of the fun of the blog post. :)
(Sometimes I get her personal email to my gmail.)
Requested my first name, it was available. I constantly got people asking questions in amazement at my email address - how did I do it??? I would just shrug and say "I asked".
Plus there is also social pressure: while you're cavorting about as an individual contributor, many other peers will be crushed under management pressure, and will start to resent you, and demand to the VPs that you share the load.
That being said, Intel is really taking a hard turn back to engineering. I might even consider going back there, assuming Gelsinger doesn't go back to the old school ranking-and-ratings that forces you up or out, or the "you must give 120%" bullshit. Being forced to work in "dungeon mode" for months at a time is what drove me out. "Dungeon mode" is supposed to be a 1-2 week thing to fix a serious bug. (DM is spending 14 hours a day in a conference room 7 days a week.) YOu can only do that so many years in a row before saying enough. That's why Intel lost so many people to Apple a few years ago.
I refuse to believe that people went from Intel to Apple for quality of life improvements
I'd think a smart move would be to hire a personnel manager for him to deal with all administrivia and let him focus on leveraging his talents with other like minds.
This is right. In most big techs, even if you manage to "technically" remain as a individual contributor at a principal/distinguished level, but you're very likely going to lead/manage a bunch of technical projects which will effectively give you tens of informal reports. Even without external social pressures, I saw many people voluntarily choose to become a manager because it's nearly impossible to handle the workload.
It's a simplifcation for sure but "autonomy, mastery, & purpose" being the elements of fulfilment rings true however hou dice this. Personally I'd really like to see a lot of really good engineers given direct unfettered access to talent, given authority while trying to minimize their personell and management overheads: let the humam resources go wild, synergize, make good things. And stand the frak back.
It's a gaping vast culutral loss that Brendan hasnt had a team of 10 under him. There are people missing from the world who would be way awesome & have done great things if Brendan had simply been tossed some people. Byt usually corporations statt having political games, attachments, commitments, things other than the work itself that comes with authority. Im fascinated by & not sure this world will ever get to grapple with just how core & essential a problem creating "low-drag" organizations is. Allowing itself to acknowledge that much of what it does is far more excellent & powerful than those in the chain of command can really.tap, harvest, understand, much less take credit for or manage.
Orgs need an anti-gravity force, a depolitical defence, of stand the f back. Brendan's go do it figure it out brilliance needs a chance & positiom in this world, needs to be shareable. Or the idea of an organization is actually a hazard, is ossified & unworthy of being the thing that has overgrown this planet loke it has. Human dignity & potential should not remain so shadowed & bound.
is anybody else bothered by the use of capitalization in email addresses? I understand that it doesn't matter semantically, but I find myself thinking negatively of people who use this for some reason.
In practice many hosted mail providers (Gmail, Yahoo, etc) treat their own accounts as case insensitive. But in MS Exchange, for example, you can have separate inboxes with only capitalization differences, so it's definitely not obscure.
> local-part of a mailbox MUST BE treated as case sensitive. Therefore, SMTP implementations MUST take care to preserve the case of mailbox local-parts. In particular, for some hosts, the user "smith" is different from the user "Smith".
Also rather OT: The Intel l219 or I219 or i219 must be one of the worst names ever for a NIC. Even Intel doesn't seem to know whether that first letter is an uppercase I or a lowercase l.
My understanding (backed up by levels.fyi) is that RSUs make up a negligible part of Netflix total comp. The deal I always heard when talking with recruiters there was that base salary was very high ($500k) but they were pretty aggressive about maintaining a churn in their employees.
But I never ended up accepting an offer there and it was awhile ago so hopefully some other Netflix employees can confirm whether or not RSUs are a major part of comp today.
When I was there the option was 20% of the stock price and you could do 100% in stock if you wanted to. So if the stock was $100 a share I'd pay $20 for the option to buy a share at $100 for the next 10 years. In other words, I was break even if the stock went up 20%, and doubled my money if it went up 40%. It was a great program when the stock was growing more than 20% a year.
From what I understand most people take all cash now, or nearly all.
Not saying the decision was caused by this. But he for sure is the "insider" that "insider trading" talks about.
Or in short, I doubt he could see this coming, especially given that his interview process had to start a few months ago.
A statement that it was not AMD would be meaningful. (We all know Qualcomm and Broadcom have their problems.)
I wonder if Brendan will directly contribute to Clear Linux. It's already the fastest Linux distro by many benchmarks. His software contribution to a distro that is focused on Intel proc would be really interesting.
Is this some sort of training/demo room?
Good luck at Intel! They do seem like a group that would be interested in the low-level optimization expertise Brendan clearly has!
Google has search.
Apple has the iPhone.
Microsoft has Windows and Office.
Amazon has AWS and logistics.
All of these are huge competitive moats.
However, unlike when Netflix first started, the moat for streaming is not the tech it is the content. As such, Disney, Universal, etc which have decades worth of movies/shows have a huge advantage in this new reality where streaming can be implemented relatively easily.
Looking at the current market capitalization of both companies, Netflix is at 84.57 billion and Disney at 203.Billion.
The difference, however, is Netflix could hone squarely in content if it needed to, and can benefit from having a single focus of mind. Disney's resources are allocated into 5 primary verticals, and 2 subsidiaries, with multiple competing budgets, priorities, resource allocations, and most debilitating at a company their size, internal politics.
That being said, Disney already funds so much of their own content generation, so the question isn't can Disney match a bid by Netflix (that's already a losing question for Disney), but can Disney create competing content that is more compelling.
And Disney had 60% subscriber growth in the past year (74M Q4 2020, 118M Q4 2021), while Netflix had 9% growth in the past year (203M Q4 2020, 222M Q4 2021), meaning the distance is shrinking as we speak. Disney sits on a massive portfolio of content, especially compared to Netflix. They are the bigger company, and have infrastructure to sell the content they own via multiple ways instead of just streaming.
The reliance on theatrical releases is a bit of a mixed bag. It is another mechanism for content generation that can add to their library, but it also comes at a loss of some value to users of Disney+ if they care about seeing stuff on release. Additionally, it's dependent on a distribution channel(cinemas) that is currently hemorrhaging money. If moviegoing doesn't recover to pre-pandemic levels before the apes' money runs out, it might prove to be a vulnerability.
the problem netflix faces is that more and more content won't be offered to netflix at any price since it'll be produced explicitly for other streaming services. If everyone has their own streaming services competing with netflix then eventually netflix is left with almost nothing but the content netflix itself produces.
Thank you for making my point?
Which ones would those be?
SVE is somewhat interesting, but I've generally found the AVX512 instructions more innovative. I really like AVX512's "compress" and "expand" instructions, for example... as well as the classic "vpermb" (but vector-permutation has been around since SSE and is an old trick: the old pshufb instruction).
Since SVE doesn't want to "set" its SIMD-width, it seems like these permute instructions (vpermb, or even compress/expand) aren't possible?
-------
I've always enjoyed Intel's innovative new instructions: PEXT, PDEP, and now AVX512 compress and AVX512 expand.
AVX512 also includes gather/scatter (but that's not innovative, been around for a long time but still nice to see it in prosumer systems)
Gather/scatter have been around for a while, but it hasn't been until more recent Intel uarch that their cost makes them worth using in practice. Zen3 is still lagging quite a bit.
I personally would have never thought of making such an instruction, despite having written multiple sets of code that use a SIMD-compress or SIMD-expand pattern.
-------
Case in point, vpcompressb (byte-wise compress) is the most blatantly obvious way to "remove redundant XML whitespace" that I've ever seen.
Its just a thing that has obvious wide-spread applicability to many algorithms I've seen and keeps coming up again-and-again. Or determining which rays (in a raytracer) are "dead" vs "alive" (separating out hits vs misses). Or implementing quicksort (compress all items "less than pivot" to X array. Compress all items "greater than pivot" to a Y array. Quicksort done).
Free-form permutation does indeed seem like less of a fit. Though it still makes sense to define a minimum vector length of N for the ISA and support permutation ops that apply the same permutation on groups of N lanes.
Also, it's an implementation choice to let you set SVE widths that aren't a power of two?