Hacker News Reader

Comment by artemonster | Hacker News Reader

This article is spot on. Verifying hardware is a much more complex task than designing it. I've spent ~1.5 years on verifying custom cpu core re-implementation (next gen) that was almost binary compatible with previous gen, we had own gcc backend and LOTS of legacy programs written in assembly. And even then it was a nightmare. Even after tons of time spent on covering most obscure cases, our FPGA platform, running our multi-task FW, would often crash: cue in hours of back-stepping FPGA trace dumps, where you have a total recording of program counter, load-store IF, internal registers for aroung 100k clocks and you have to trace back the execution and see where the fault occurs. Usually it would turn out to be some weirdly obscure combination of recently-fired interrupt, some pipeline stall with execution of some weird instruction. fixed in 3 seconds when found. goddamn that was exciting. I wonder if there are any people here on HN that work on advanced HW verification techniques that are currently ongoing in the industy?

View on HN

Yep — and I’ve already said too much. I get upset how secretive this industry is; I truly believe it’s at least 30 years behind the curve (and falling further) compared to SW. I also firmly believe it is the EDA vendors who sell this mindset because $$$.

this industry is a Borland's wet dream - everyone uses crappy expensive software that reeks of 90s, the libraries are closed source with barely exposed interfaces, the language is a mess (SV), etc. Yes, you are right, as long as its making them huge money nothing will change. And nothing can change, since the field is a frickin mine field of patents, you cannot innovate. Also, why you've said too much? What have you said? :)

sounds like it's well functioning capitalist market.

lots of money is made by a small group of very smart people in an entrenched position.

Please do not take HN threads on generic ideological tangents. It leads to generic flamewar hell. Regardless of ideological preference, none of us want to end up there.

https://news.ycombinator.com/newsguidelines.html

If the low quality of the software impedes productivity generally (and, being honest, it MUST) then that’s not well-functioning. It’s more a common depressing failure of a capitalist market (and a lot of software verticals suffer from it).

It’s a space with a high barrier to entry, which is probably the biggest problem. Software engineering tools got good, and affordable, due to a combination of lowish barrier to entry and open source.

so it sub-optimizes?

capitalism is at its best when optimizing industrial production (manufacturing materials)

knowledge creation is not quite an industrial process.

I would highlight the importance of open source as a key component for better tooling.

But knowledge does not quite play well with capitalist markets. there must be a better way, but finding it requires revising several 'fundamental' assumptions about how this (market) society works.

As a hardware designer (well, mostly a hardware verification engineer), I feel that this kind of comment is a pure misconception from software people that gets echoed time after time in HN.

I generally see a lot of more effort and good quality tools in hardware verification than in software validation. The hardware design and verification industry is not really 30 years behind the curve compared to software development.

Hardware and software are two very different problems with their own set of constraints. It's actually more often that you see software barely tested being deployed in the wild to millions of users rather than hardware. Hardware bugs tend to get more attention because you can't really deploy a patch to fix the issue.

> I feel that this kind of comment is a pure misconception from software people that gets echoed time after time in HN.

but this is NOT a misconception. Show them actual tools that constantly crash and lag on you, that demand checking out licence for a runtime which costs 5-100k/Year, show them environment in which we have to write our code - they will laugh at you. This was exactly the state of the software industry 30 years ago and this industry is long past that stage.

Those are some very empty counter arguments but I will try to answer them anyway.

EDA tools are expensive, yes. But you are not just paying for the tools, you are paying for all the engineering support being those tools. Bugs and problems happen in any development tool and those companies are ready to pick up the phone at any time of the day to help you with anything you might face, whether it is a problem with their tool or a problem with their setup, to the point that they can deploy a new version of their software on the spot just for you.

And they have actual engineers on the other end of the line who know the product like the palm of their hand.

While this might sound superficial, it is extremely important when you are racing towards a $100M dollar deadline for your project.

> EDA tools are expensive, yes. But you are not just paying for the tools, you are paying for all the engineering support being those tools.

So I can buy the tool without the support contract?

> Bugs and problems happen in any development tool [...]

So it's automatically a wash? Why bother trying to improve things if there will always be bugs, right?

> [...] whether it is a problem with their tool or a problem with their setup, to the point that they can deploy a new version of their software on the spot just for you.

I'm not impressed. This is a kludge.

> And they have actual engineers on the other end of the line who know the product like the palm of their hand.

Okay, that actually sounds pretty nice.

> So I can buy the tool without the support contract?

Probably not but it's not something you would like to. Cutting corners is a really bad idea when it comes to hardware development because once it is out there, there is not going back.

Having access to a professional engineering work force that knows how the tool works and know it will ready and interpret the Verilog and process it can make a difference between a 5 minutes delay and 2 week delay in the project.

> So it's automatically a wash? Why bother trying to improve things if there will always be bugs, right?

You are distorting my words. I'm not saying that we should be complacent and just accept incompetence.

>> [...] whether it is a problem with their tool or a problem with their setup, to the point that they can deploy a new version of their software on the spot just for you.

> I'm not impressed. This is a kludge.

So how would you handle it? Tell the customer you'd love to fix their problem but they have to wait for the next formal release?

The level of quirkiness and bugginess in the tools is on another level to most software tools. You shouldn't need access to the engineers who built it in order to make it work, this basically increases costs and barrier to entry massively.

Many of the tools are bleeding edge, developing new features to support a new process node at the same time as the new process node is being developed.

This doesn't seem relevant to most of the tooling, which is not bleeding edge, and should be mature.

Certainly on the simulator/synthesis side you can still encounter straight-forward bugs in SystemVerilog features that were standardized over a decade ago (or find they're not implemented at all) or even simple parsing bugs (the kinds of bugs that would be trivial to find with some decent randomised testing of the parser).

Things being bleeding edge and fast evolving in certainly true for some EDA tools or parts of them but there's lots of more bread and butter stuff that never feels quite right.

That is no excuse. A lot of software out there is bleeding edge, yet costumers do not tolerate crashes.

Sure, that's why important software never crashes. /sarcasm

And... it does not. Most crashes in actually important software are due to hardware/memory errors, rather than the software itself.

Of course, I hope you are not referring to Windows or any day-to-day software by "important".

> The level of quirkiness and bugginess in the tools is on another level to most software tools. You shouldn't need access to the engineers who built it in order to make it work, this basically increases costs and barrier to entry massively.

I do not know what SW tools are you using but i'm still looking forward for a SW tool that doesn"t suck.

Oh yeah, software tooling also sucks, I'm just saying it's on average still better than the best of any hardware tooling I've ever had to use.

I believe software developers are split between IntelliJ, Eclipse, Visual Studio and VS Code. They do work very well compared to what I've seen in the embedded world.

Ignoring text editors here, because text editors are not programming environments (no debugger, no compiler, no source control, etc...)

I mean, this sounds a lot like software 20 years ago.

You can still get proper support for, say, IntelliJ if you want it, but tools and ecosystem have improved so much that it’s just not as important as it used to be.

To be clear, I’m not claiming that hardware development is the same as software development. It’s not. It has different problems. But the tooling _does_ seem well behind.

> You can still get proper support for, say, IntelliJ if you want it, but tools and ecosystem have improved so much that it’s just not as important as it used to be.

How many people are doing hardware verification work vs. cranking out webapps with IntelliJ?

JetBrains can invest a lot in software ergonomics because they're basically selling mass-market software. If you have sophisticated but niche software, your probably going to invest your engineering effort into the it's capability rather than its ergonomics.

I mean, to take one example of a company in the space: Xilinx has operating income of about 750mn per year. Jetbrains has operating income of about 80mn per year on revenues of 250mn.

Given that I’d suggest that the reason that Xilinx’s development products have such a poor user experience vs IntelliJ is NOT that Xilinx is impoverished, but rather that the market has extremely low expectations so they don’t feel the need to spend on fixing them.

This isn’t a problem unique to hardware-land; lots of verticals have incumbents who produce low quality products on large profits because, frankly, the competition is just as bad and there’s a high barrier to entry. I’m not sure what would fix this for hardware-land, though. Possibly more pressure from open source; that’s largely what did it for software development (though even before that, companies like Borland put more than the minimum effort in).

Xilinx are not just writing software, they also must use R&D to develop the FPGA hardware, dev kits, IP, etc, so it's a very different scale to JetBrains' focus. Also in terms of Sofware, the really challenging aspects are under the hood, the place and route algorithms, the static timing engine, the system verilog simulator (which is itself a huge undertaking), the hardware debug cores, the HLS SW that compiles C to accelerated HW, then SDK for embedded processors, oh.. And of course... Finally the GUI/IDE. With all that's going on underneath, Vivado does a decent job of presenting it all in a logical way. I'm blown away that I can see a timing failure on any of a million nets and in a click cros probe it to a line of verilog.

As for open source, SystemVerilog is an open standard, yet can you show me an open source simulator that competes with the free one that comes with Vivado?

On the one hand Vivado is not entirely terrible considering that it can get some work done, but on the other hand it's beyond me why it's lacking the most basic needs of software development.

Zero support for source control? SoC have no examples, why can't they systematically provide an hello world example (blink a LED) with their development boards? The IDE itself has close to zero doc and there is no help on the internet, it takes 3 freaking days for an experienced developer to figure out how to create a project.

Theres a load of documentation, videos and examples, both on github maintained by xilinx as well as on xilinx.com. For example: https://www.xilinx.com/support/university.html and linked from there is a complete course in getting started with all materials, slides and docs : https://www.xilinx.com/support/university/vivado/vivado-work...

Have you used software verification tools? They're almost universally terrible. In many cases incomprehensibly so. Many are free up front, but you pay for it one way or another. The level of sophistication and workflow integration of software verification tools, compared to hardware verification tools, also seems to lag significantly.

I don't doubt that the hardware verification tools are expensive, buggy, and probably have terrible UX. However, all those things are also true of software verification tools, but where license cost is replaced by esoteria and incompleteness cost.

Valgrind is pretty robust and I've seen it integrated into many projects with little fanfare. Unit tests are just simulation runs.

I've done VHDL/Verilog and tons of software, the HW tools do lag significantly in my experience.

Valgrind is not in the class of verification tools I was referencing. How does it correlate to VHDL/Verilog in this context? I would expect things like MBD tools, model checkers, and proof assistants, and program extractors to be the software pipeline siblings to the kinds of tools that VHDL/Verilog integrate with.

It's a tool to verify the software runs correctly and doesn't double free, write over the heap and a while bunch of failure. Unless you never run simulation and always trust the router?

Not everything gets verified 100% in HW, otherwise we wouldn't have silicon errata. It's all a question of how deep you want to go in verification and what your costs are for screwing it up.

Having worked with both HDLs and software tooling, I don't think it's too far off the mark. HDL tooling is terrible software: it's hard to use, buggy, expensive, opaque, and the support is terrible unless you're a huge customer. Hardware verification and development works still through sheer effort, and because there's a strong business incentive to get things right the first time due to the huge costs of a respin. With better tools this could be done faster and cheaper with similar reliability, but perhaps there isn't enough to be gained to make it worth the effort (and either way I don't think the companies currently working on the tooling are capable of making anything else: this would need disruption to really make a difference, and the barriers to entry are huge).

We get decent support from our main tooling providers. We do spend a chunk on licenses but I don't think we'd be considered a huge customer. And yes, the software has bugs (with expensive consequences) so good support is very important.

Hardware design flows are complex and do have bugs, but not in the way it sounds. In general HDL synthesis produces a design that matches what the user wrote, so a logic bug in a synthesis engine is rare. The bugs exist in the 'other stuff', like trying to convince the tool to be efficient about how it created the design, or smart about how it's trying to close timing. Getting a working though the tools design is easy, getting an optimal design is hard. HDLs themselves have issues, but there is a wealth of quality checking tools built around them that mitigate them, and indeed mitigate/flag poor coding from inexperienced designers (to a degree). What's interesting is FPGA tools, as much as they are derieded, are much better than their Asic counterparts in terms of functionality and user friendliness.

> I feel that this kind of comment is a pure misconception from software people that gets echoed time after time in HN.

But the comment came from a hardware designer, at least the grandparent one (from artemonster)?

> I feel that this kind of comment is a pure misconception from software people

I really hoped you were referring to the secrecy

Yes, hardware must be right first time. Proving that is difficult, so logic equivalence, formal proof, code coverage are all standard. What's optional in many software QA flows would be absolutely required in HW flows. Proof of functionality is paramount

I don't agree. I agree that if compare random software project x to a random hardware project y that the hardware project will do a lot more verification.

But that doesn't really compare to state of the art of what's possible in software. Property based testing, symbolic execution etc all have incredible power that afaik is not available in the hardware world. I'm just a hobbyist who uses FPGA's but it's way way easier for me to ensure a function has some properties compared to a VHDL module or whatever.

> 30 years

Well, microcircuits are also subject to physical constraints that have been around for almost 14 billion years...

So are algorithms and software, except the rules might be even older

Ah yes. Because software is not subject to physical constraints.

Judging by today’s software, looks like no, it’s not.

I haven't seen a program solving the halting problem. Because you know physical constraints exist.

Not only that, but the sheer complacency of people within the industry when it comes to better tools blows my mind!

Our VDI development environment runs an OS from 2 decades ago. It's ugly and the latency is mildly infuriating. And yet, my co-workers are totally OK with it all.

I've been working for the last four years as a consultant for a chip design company on one of their internal design tools. The complexity is absolutely mind-boggling. And seeing how the sausage is made doesn't make me any less queasy about it.

I have been there, it's horrible. Often traces are in binary, and I spent time making them more readable, to take off the intellectual load of just decoding what you see from the actual debugging.

These days I design Verilog for FPGAs without a verification department. I've learned to code defensively, to use what I know is robust without creating complex corner cases. This often involves clear handshaking and dataflow. Anyway, it's more fun than fixing bugs deep in the weeds of a complex system.

I work in processor verification and my employer spends a great deal of money on it. Like "thechao" I am not allowed to talk about this, alas. A good place to look at some of what ARM do is:

    https://alastairreid.github.io/

The secrecy is pretty effed up.

CPUs are arguably the most crucial mechanical invention in history and yet the practical art of making them actually work right is shrouded in secrecy in service of greed.

On Windows/MacOS, the communication between the operating system and external devices (drivers) are completely shrouded in mystery. Theoretically the Nvidia graphics driver could be intercepting all your ethernet packets and sending them to the US government. (I have no evidence of such a attack, but it is theoretically possible) Drivers are extremely privileged, and are also a major gateway towards compromising computer systems)

That is not really comparable. If someone murdered all of the GPU driver writers working or retired, we'd have new drivers in maybe 1 year? 2? 5?

On the other hand, if someone were to murder all of the currently working or retired processor designers, we'd probably have new i7s in a few decades.

If you don't think the GPU drivers are complex, then look at the perf difference between nouveau and the nvidia proprietary driver. Those guys are also working on the bleeding edge of optimization. Sure you could write a working driver in short order, but its not going to perform the same.

I'm not saying that they are not complex, that's why I chose them. But you have the source code for them, you don't have to write them from scratch. Some of the optimization techniques, even if not documented, can probably be studied from the change history. Of course, it will take a few years to grow new experts without knowledge passing from the old ones, but by and large the logic is there in the code.

From what I understand, the same can't be said of real microprocessor design - the complexities of aligning logical requirements with EE complexities, analog responses, and manufacturing process limitations are not at all captured in the 'code', not at the same level. We can hope that they are documented, at least to some extent, but we all know the general priority of internal documentation against other concerns, and the difficulty of documenting design processes.

This may be true in some cases, but from my experience (nearly 20 years in custom IC design) good design is still more difficult than verification. Verification seems to have an element of over complication. I've seen simple, totally independent sub components take many man weeks of verification, not because it needed to, but because the verification approach, even for simple things can be overly complex. So sometimes the reason verification can take more resources is not because its harder, its because the approach for that particular problem was not efficient. Of course like anything there are extremes on bot sides - but good design must be both correct, efficient, and meet timing/power goals, while verification must just be correct.

we had own gcc backend

Brings me back to 1995 when I was working on the C-Cube CL4010 MPEG-2 encoder processor. A fully custom 32-bit (with some 36-bit registers) RISC architecture.

http://www.w6rz.net/cl4010.png

> This article is spot on. Verifying hardware is a much more complex task than designing it.

Are we sure about that? They start out talking about how hard it is to write correct state machines in Verilog, and they follow that with a lot of skepticism about RISC-V processor designs. But most RISC-V designs use Chisel, not Verilog. Chisel is much higher level than Verilog and using it should also make it quite a bit easier to prevent many design-level errors.

Chisel is not a magic bullet and does not make you suddenly write bug-free hardware. Its the same logic applied when people compare coding in x vs Haskell. The latter does not suddenly grant you superpowers to write bug-free software.

>Chisel is much higher level than Verilog

have to disagree here. the level of abstraction is exactly the same in chisel as in verilog, its still RTL level. High-level abstraction in hardware is a long-chased holy grail that is revived in the industry every now-and-then and then immdediately dismissed and forgotten, when faced with real-world industry challenges.

No is claiming that Chisel code is always bug free. Just like no is claiming Haskell code is bug free. However the point they are making is that these languages are way better in statically catching a whole host of bugs. They also allow certain properties to be expressed in the type system which is essentially having a proof that that invariant holds now matter what.

I don't think that is the point of the article. Even a state machine that is very well written can be wrong if it's fed with the wrong input. And this is the core of the problem with hardware development.

You want to make sure that the hardware you are designing never ends up in a situation that you didn't foresee. So you want to make sure that during hardware verification you go through every single possible combination of inputs and you want to make sure you exercise every single possible outcome.

The quality of the hardware is not about the language you write your hardware in. It's about the tools, the methodology and the infrastructure you use to exhaustively test your hardware.

And this is the point of the article, current processors are so complex and do so many things that the hardware verification suffers from an explosion of states to exercise.

On the other hand it's a repeating problem. We've being doing processor verification for a long time.

Why can't we reuse that knowledge, that code ?

It's not the same thing. Chisel will help you do things like write more reusable RTL, because the language itself is more expressive -- so the potential level of abstraction is much higher than what you can achieve in Verilog. You can do more flexible kinds of parameterization, for example. And doing that can help avoid certain kinds of bugs, for sure. (And it also means you simply need less RTL to describe the same thing, because re-use is so much higher.)

But Chisel will not help you when you have some errata like "the BRAMs on these devices take 67 cycles to initialize after coming out of reset" tucked away in a manual somewhere that you forgot to read. That's just an FPGA example -- you can run a hundred simulations and your design will still fail immediately in real hardware because of such things. (Even "cycle accurate simulations" from your vendor may not account for these things.) You're going to encounter a lot of problems like this, before you even move to ASIC level flows.

Ultimately the hardware industry does need much better RTLs, because the current ones mostly suck from a language POV in a huge number of ways. But verification is still a much bigger problem in general no matter what RTL you've chosen.

   tucked away in a manual

With a better eco-system, the (temporal) specification (such as "take 67 cycles to initialize after coming out of reset") such be given formally, so a tool can ensure that it's not forgotten. In principle this is possible.

   much better RTLs

Have you got any concrete ideas and proposals? I'm asking because (A) I agree with you, and (B) I'm very much in a position where I can influence research in this direction.

> (B) I'm very much in a position where I can influence research in this direction.

Could you elaborate on what you do?

> Have you got any concrete ideas and proposals?

My recommendation is: try to build something complex, or take something from open-source that is very complex and extend it (test it, verify it, improve performance, etc.). Notice the problems you encounter, and then try to solve those problems.

I could give you a laundry list of pain points myself (why do I have to pay money for lint, and why does lint still suck???), but I think the best research is done by people trying to solve problem X, but along the way ended up having to solve Y and Z just to get to X.

I've spoken to a lot of processor designers, usually from an EE background, and they prefer Verilog: "we tried more high-level languages in the 1980s ... it was a failure" is something I've heard more often that than I wish I had.

I think there is an interesting C. P. Snow-like Two Cultures thing at play between SW and HW people. I think the latter don't quite get the abstraction power of modern programming languages, they don't understand that syntax, types, modularity matter. They think PLs are something like C and C++: a mess that we simply accept and get on with life. The former really underestimate the extreme scale and complexities involved in processor design and verification. Never the twain shall meet?

   laundry list of pain points

Think bigger: what would you do if you have a team of 3 engineers, or 30 engineers to work on a better tooling ecosystem?

So...

Doing a complete verification of a CPU in a strict sense is impossible.

These days we are relying more and more on computational proofs for math and science research.

Therefore - what can we say about the solidity of our scientific knowledge going forward?

It sounds like hubris, y'all are trying to bat above your league.

Metaphorically, it's like the rush to build auto-autos (self-driving cars): the problem is too hard. If they had started by trying to make a self-driving golf cart Elaine Herzberg might still be alive.

If "verifying hardware is a much more complex task than designing it" (and I don't doubt it) then that is the limiting factor. (Or should be IMO. The liberties taken by software cowboys are bad enough w/o the hardware getting all squirrelly too.)

Yeah I agree. In my opinion the consequences are showing now that all processor designs are suspect and likely incompatible with the very real threat models we face day to day when we browse the untrusted web.

Reply on news.ycombinator.com