Shift Left Is the Tip of the Iceberg
semiengineering.com
semiengineering.com
[1] https://en.wikipedia.org/wiki/Shift-left_testing [2] https://www.dynatrace.com/news/blog/what-is-shift-left-and-w...
Source: the week I spent in the Valve offices in 2005, having Valve staff playing our in development HL2 mod.
Other examples:
* Replacing automated tests with (quicker) type checking and running it on a git commit hook instead of CI.
* Replacing slower tests with faster tests.
* Running tests before merging a PR instead of after.
* Replacing a suite of manual tests with automation tests.
etc.
- implementing security features earlier (DevSecOps)
- implement tracing and metrics/analysis tools earlier, use them to test and debug apps earlier (as opposed to laptop-based solutions)
- building the reliable production model earlier (don't start with a toy model on your laptop if you're gonna end up with an RDS instance in AWS; build the big production thing first, and use it early on)
- add synthetic end-to-end tests early on
The linked article is talking about Shift Left in the context of developing semiconductors, so you can see how it can be applied to anything. Just do the needed thing earlier, in order to iterate faster, improve quality, reduce cost, ship faster.
But with chip design, they can’t iterate that fast and performance is more important, so they are doing more design and testing before the expensive part, using increasingly elaborate simulations.
There's no need to write tests upfront for you to shift left. All shift left means is that testing happens during development. Whether you start by writing tests and then write the actual program or the other way around -- doesn't matter.
Nobody I've worked with can ever quantify the ROI for elaborate take test environments, but somebody made an okr so there you go. Far be it we follow actual research done on modern software... http://dora.dev
I'm firmly of the opinion that if a test can't be run completely locally then it shouldn't be run. These test environments can be super fragile. They often rely on a symphony of teams ensuring everything is in a good state all the time. But, what happens more often than not, is one team somewhere deploys a broken version of their software to the test environment (because, of course they do) in order to run their fleet of e2e tests. That invariably ends up blowing up the rest of the org depending on that broken software and heaven help you if the person that deployed did it at 5pm and is gone on vacation.
This rippling failure mode happens because it's easier to write e2e tests which depend on a functional environment than it is to write and maintain mock services and mock data. Yet the mock services and data are precisely what you need to ensure someone doesn't screw up the test environment in the first place.
Obviously this is for large-scale systems and not small teams.
Personally I think the real issue is not the testing strategy but the system itself. Many organizations make systems overly complex. A well structured monolith with a few supporting services is usually easy to test while micro service/SOA hell is not.
I heard a story decades ago about a software team that got a new member transferred in from the IC design department. The new engineer checked in essentially zero bugs. The manager asked what the secret was, and the new engineer said “wait, we’re allowed to have bugs?”
If you could spin a chip every day that would mostly be a huge waste of time.
From a QA perspective, I greatly regret that the world of infrequent releases is mostly gone. There are few kinds of products that still hold onto the old strategy, but this is a dying art.
I see the world of services with DevOps, push on green etc. strategies as a kind of fast-food of software development. A kind of way of doing things that allows one to borrow from the future self by promising to improve the quality eventually, but charging for that future improved quality today.
There are products where speeding the rollout is a bad idea. Anything that requires high reliability is in that category. And the reason is that highly reliable systems need to collect mileage before being released. For example, in storage products, it's typical to have systems run for few months before they are "cleared for release". Of course, it's possible to continue development during this time, but it's a dangerous time for development because at any moment the system can be sent back to developers, and they would have to incorporate their more recent changes into the patched system when they restart the development process. I.e. a lot of development effort can be potentially wasted before the system is sent out to QA and the actual release. And, to amortize this waste, it's better to release less frequently. It's also better to approach the time when the system is handed to QA with a system already well-tested, as this will minimize the back-and-forth between the QA and the development -- and that's the problem shift-left was intended to solve.
NB. Here's another, perhaps novel thought for the "push on green" people. Once it was considered a bad idea for the QA to be aware of implementation details. Testing was seen as an experiment, where the QA were the test subjects. This also meant that exposing the QA to the internal details of the systems or the rationale that went into building it would "spoil" the results. In such a world, allowing the QA to see a half-baked system would be equivalent to exposing them to the details of the system's implementation, thus undermining their testing effort. The QA were supposed to receive the technical documentation for the system and work from it, trying to use the system as documented.
Start paying "QA" more than their dev partners consistently and with better promotion opportunities and you can get better testing, but everybody seems to be making plenty of money without it.
From the perspective of a software user, I greatly regret the same thing. I really think that rapid/continuous release has done more harm than good.
It baffles me that anyone would continue to promulgate the Pressman numbers (which claim ~exponential growth in cost) based on... it's not entirely clear what data, as opposed to Boehm's paper which only claims a linear relative cost increase, but is far more credible.
In a waterfall, single-deliverable model it wouldn't surprise me there is some increase in costs the later a bug is discovered, but if you're in that world you have more obvious problems to tackle.
People still use the Pressman numbers. So much for "data-driven decision making"...
”prevention is better than cure”
[ante-] ”closing the stable door after the horse has bolted”
One nice side effect of tying the mnemonic to reading direction rather than homonyms is that it carries over across languages better (though still imperfectly).
Maybe everything about those concepts are just wrong? I mean, you can have people like Uncle Bob who’ll tell you that “you just got it wrong”. He’s also always correct in this, but if so many teams “get it wrong” then maybe things like clean, xp and so on simply suck?
It's worth noting that left-leaning systems were widely used once too, and they had about half of the overall success rate of the right-leaning ones on those criteria.
From my anecdotal experience YAGNI and making sure it’s easy to delete everything is the only way to build lasting maintainability in software. All those fancy methods like SOLID, DRY, XP are basically just invitations for building complexity you’ll never actually need. Not that you can really say that something like XP is all wrong, nothing about it is bad. It’s just that nothing about it is good without extreme moderation either.
I guess it depends on where you work. If it’s for customers or a business, then I think you should just get things out there. Running into scalability issues is good, it means you’ve made it further than 95% of all software projects.
The classic example is driver development. No one, even today, sits down and writes e.g. a Linux driver until after first silicon has reached the software developers. Early software is all test stuff, and tends to be written to hardware specs and doesn't understand the way it'll be integrated. So inevitably your driver team can't deliver on time because they need to work around nonsense.
And it feeds back. The nonsense currently being discovered by the driver folks doesn't get back to the hardware team in time to fix it, because they're already taping out the next version. So nonsense persists in version after version.
Shift left is about trying to fix that by bringing the later stages of development on board earlier.
We used simulators for hardware. Using parts of existing hardware when possible and hacked Microsoft sw emulation to behave as our hardware would.
When the hardware came back, it was less than a day to get the driver running.
So you make a tool that prevents errors and speeds up a process, now people use it four times as much, and now they wonder why they're only seeing half as many faults instead of an order of magnitude less.
We are humans. We cannot eliminate errors, we can only replace them with a smaller number of different errors. Or as in your case, a larger number of them.
Isn't ignoring the early steps that could save time later also known as false economy?
It's only a false economy if they are the correct steps. If it turns out that they are wrong, it's a very real one.
I have always called this "front loading" and it's a concept that's been around for decades. Front loading almost always reduces development time and increases quality, but to many devs, it feels like time-wasting.
So before we clearly clarify product requirements, we start the build early with assumptions that can change. Sometimes it works, sometimes it does not, which just means we end up net neutral.
But an executive somewhere up the management chain can claim more productivity. Lame.
[1] I work at a bank.
I consider it a success when an entire line of work is scrapped because after initial testing the users say they wouldn't use it. Depending on the project scope that could be 6-7 figures of dev time not wasted right there.
"Build fast and break things" works in big tech, but its a serious risk in financial services.
That being said, we are being forced to "build faster and try not break things."
I agree with the previous commenter. It isn’t about breaking things out even moving fast (in my experience), but about getting “the juices flowing”.
I’ve found that when there’s a request for some new tool or thing, people have a very difficult time defining [all] the real world needs. They know they need X, but don’t remember - or even realize - they also need Y, Z, etc. Or worse, many people have been conditioned through corporate culture that they can’t even ask for things that would make their lives easier.
Getting something basic in someone’s hands quickly with the desire of creating a loop of feedback, feature, feedback, … gets the end user more focused and engaged with the final product, which gives the engineers more focused requirements. I’ve found it also gives the engineers the opportunity to make suggestions and get more invested.
We respond to spikes in increased exceptions, then read logs to try to figure out why something "went wrong". If we can change some code to fix it, we do so. But since we don't have confidence in our fixes, we have to move further back to the design and guess how the system is supposed to work. I'm hoping next month we start a dialogue with with upstream and ask how we're supposed to use their API.
We seem to be hell bent on ignoring age old common sense repeatedly in our work while simultaneously inventing new names for them. Wonder why? Money? Survival? Personal ambition?
These concepts are not worth the paper you wipe your backside with unless you have a manager and a team that cares.
The main benefits you get from this is reduced context switching, increased information retention, increased ownership.
But it has to go with tooling, and process that enables that. If you have a really long deployment process where devs can get distracted then they will lose context switching between tasks. If you make every Dev a one man army that has to do everything on their own you won't be able to ship anything and your infra will be a disjointed mess.
They key thing is reducing toil, and increasing decision making power within a single team/person.
From the sound of the article they might be just squishing two teams together? What's the advancement that made the two steps be able to happen at the same time?
Tell me you work for Intel without telling me you work for Intel.
> says Chris Auth, director of advanced technology programs at Intel Foundry.
Yeah that’s what I thought. The breathlessness of Intel figuring out things that everyone else figured out twenty years ago doesn’t bode well for their future recovery. They will continue to be the laughing stock of the industry if they can’t find more self reflection than this.
Whether this is their public facing or internal philosophy hardly matters. Over this sort of time frame most companies come to believe their own PR.
You can absolutely shoot your feet off trying to modernize too quickly. Intel will be the laughingstock if 18A never makes it to market and their CPU designs start losing in earnest to their competitors. But right now, in a relative sense, Intel isn't even down for the count.
And that’s just the current product! The last two gens are unreliable, quickly killing themselves with too high voltage and causing endless BSODs.
The culture and methods of ex-Intel people at the management level is telling as well, from my experiences at my last job at least.
(My opinions are my own, not my current employers & a lot of ex-Intel people are awesome!)
A lot of hopes and promises are riding on 18A being the savior for both Intel Foundry Services and the Intel chips wholesale. If we get official confirmation that it's been cancelled so Intel can focus on something else then it will signal the end of Intel as we know it.
The next 3 years, 2021/2022, 2022/2023 and 2023/2024, have been dominated by "10 nm" rebranded as "Intel 7" products, which have used the Golden Cove/Raptor Cove and Gracemont CPU core micro-architectures.
Now their recent products are split between those made internally with the Intel 4/Intel 3 CMOS processes (which use rather obsolete CPU cores, which are very similar to the cores made with Intel 7, except that the new manufacturing process provides more cores per package and a lower power consumption per core) and those made at TSMC with up-to-date CPU cores, where the latter include all the higher end models for laptop and desktop CPUs (the cheapest models for the year 2024/2025 remain some rebranded older CPU models based on refreshes of Meteor Lake, Raptor Lake and Alder Lake N).
Really? I thought they said using e-cores would be better than hyper threading. AMD has doubled down on hyper threading - putting a second decoder in each core that doesn't directly benefit single thread perf. So Intels 24 cores are now competitive with (actually losing to) 16 zen 5 cores. And that's without using AVX512 which Arrow Lake doesn't even support.
I was never a fan of big.little for desktop or even laptops.
(Wikipedia seems to want me to call it "heterogeneous computing", but that doesn't make sense - surely that term should mean running on CPU+GPU at the same time, or multiple different ISAs.)
Of course, it might've worked fine if they used symmetric CPU cores as well. Hard to tell.
Because it's an ARM trademrk, and that's how they want it written:
https://www.arm.com/company/policies/trademarks/arm-trademar...
> (Wikipedia seems to want me to call it "heterogeneous computing", but that doesn't make sense - surely that term should mean running on CPU+GPU at the same time, or multiple different ISAs.)
According to Wikipedia, it means running with different architectures, which doesn't necessarily mean instruction set architectures.
They do actually have different ISAs though. On both Apple Silicon and x86, some vector instructions are only available on the performance cores, so some tasks can only run on the performance cores. The issue is alluded to on Wikipedia:
*> In practice, a big.LITTLE system can be surprisingly inflexible. [...] Another is that the CPUs no longer have equivalent abilities, and matching the right software task to the right CPU becomes more difficult. Most of these problems are being solved by making the electronics and software more flexible.
No, there's no difference on Apple Silicon. You'll never need to know which kind of core you're running on. (Except of course that some of them are slower.)
I don’t know AMD well enough to say whether it works better for them.
There is one important application for which SMT is almost always beneficial: the compilation of a big software project, where hundreds or thousands of files are compiled concurrently. Depending on the CPU, the project building time without SMT is usually at least 20% greater than with SMT, for some CPUs even up to 30% greater.
For applications that spend much time in executing carefully optimized loops, SMT is usually detrimental. For instance, on a Zen 3 CPU running multithreaded GeekBench 6 with SMT disabled improves the benchmark results by a few percent.
This is reality in VLSI CAD.
Now you understand why everything in hardware engineering is stupidly dysfunctional.
We don't need "shift left". We need tools that don't cost a megabuck.
The existing open PDKs are for technologies from 20 years ago.
If TSMC, Intel, Samsung, or even Global Foundries or other second tier foundry would publish the design rules for their manufacturing processes and specifications of the formats for the product documentation that they must receive from a customer, it would be easy to make something better than the overpriced commercial tools.
I have used for many years the tools provided by Cadence, Mentor and Synopsys, so I know with certainty that it would be easy to improve on them if only one would have access to the documents kept secret by TSMC and the like.
This collusion of the foundries with the EDA vendors that eliminates the competition from the EDA market does not make much sense. Normally better and cheaper design tools should only bring more customers to the foundries, so I do not understand what do they gain from the status quo. (Unless they fear that public documentation could make them vulnerable to litigation from patent trolls.)
Fear of competition in the foundry market cannot be the reason for secret design rules. There is negligible competition in the foundry market and the most dangerous competitor of TSMC is Intel. Yet TSMC makes now chips for Intel, so the Intel designers have seen much of the TSMC design rules and they could chat about them with their buddies from the Intel foundry division. So TSMC must not give great importance in keeping the design rules secret from direct competitors. Therefore they must keep the rules secret mainly for other parties, but I cannot guess who are those.
What the foundries gain isn't clear, but they have more customers lined-up than they can serve, so they don't lose anything to it.
I am not sure that they have more business than they know what to do with, esp at larger node sizes. Most foundries are forever stuck at what their current node is. 40nm is still plenty good for the majority of long tail ICs.
The problem is that we don't really need more digital. Digital is covered--there is a 99% probability that you can abuse somebody's microcontroller to do what you need if what you need is digital.
That isn't true if your need is analog or RF. If the thing you need is even slightly off the beaten path, you're simply out of luck.
The big problem with those right now is DRC (design rule checking), LVS (logical vs schematic), and parasitic extraction. Magic wasn't even good back in the 90s, and technology moving forwards has simply made it even less good.
The issue is that funding for the open source EDA stuff mostly comes from the US DoD, and the US DoD is only funding all this because they want to somehow magically make digital hardware design a commodity so that their projects somehow magically get cheaper.
The problem, of course, is that the primary reason US DoD chip runs are so damn expensive is that they are stupidly low volume so all the NRE costs dominate. Even worse, everybody in government contracting knows that your funding is whimsical, so you frontload the hell out of everything to minimize your funding risk. So, you have big upfront costs multiplied by a big derisking factor.
The real solution would be for the DoD to pull this stuff in house somewhere--software, design and manufacturing. But that goes against the whole "Outsource All The Things(tm)!" mantra of modern US government.
Difficulty: Poster runs Santa Claus's data center.
Not really. Magnetic declination is a thing. And besides, unless we're standing in the same place, my "north" is not your "north." If you want to fight me on that I'll meet you one degree due north.