Nyuzi – An Experimental Open-Source FPGA GPGPU Processor
github.com
github.com
[1] https://github.com/elec-otago/openvga
I think the most useful thing right now would be a high quality version of the "easy" parts of a GPU. Basic scan out, possibly overlays, color space conversion, buffer handling. This would allow ANY open processor projects to have frame buffer graphics and run LLVMpipe for basic rendering and desktop compositing. This may be slow, but it is required for every open GPU project, while a SoC can live without the actual GPU for some applications.
IMHO, first thing first.
i was stunned to find that in the hardware world, test-driven development is not standard practice: people simply haven't been trained that way. they write thousands to tens of thousands of lines of unverified code and finally do an integration test.
* litex (choose any one of the available cores)
* richard herveille's excellent rgb_ttl / VGA HDL https://github.com/RoaLogic/vga_lcd
* some sort of "sprite" graphics would do https://hackaday.com/2014/08/15/sprite-graphics-accelerator-...
the real question is: would anyone bother to give you the money to make such a project, and the question before that is: can you tell a sufficiently compelling story to get customers - real customers with money - to write you a Letter of Intent that you can show to investors?
if the answer to either of those questions is "no" then, with many apologies for pointing this out, it's a waste of your time unless you happen to have some other reason for doing the work - basically one with zero expectation up-front of turning it into a successful commercial product.
now, here's the thing: even if you were successful in that effort, it's so trivial (Richard Herveille's RGB/TTL HDL sits as a peripheral on the Wishbone Bus) that it's like... why are you doing this again?
the real effort is the 3D part - Vulkan compliance, Texture Opcodes, Vulkan Image format conversion opcodes (YUV2RGB, 8888 to 1555 etc. etc.), SIN/COS/ATAN2, Dot Product, Cross Product, Vector Normalisation, Z-Buffers and so on.
We need HDMI output, preferably 4K capable. I also mentioned colorspace conversion. Should have also said to "just throw in" video decoder for VP9 and AV1 if that's available. The point is that the likes of SiFive and other Risc-V SoC vendors should be making desktop chips, not just headless Linux boards or ones with proprietary GPUs.
Like I said, the "easy" part should be done and available - not theoretically assemblable from various pieces.
If this were readily available, I'd be able to buy it from someone today. There IS a market for it and that will be growing fast. Add a real GPU and things look even better.
this can entirely be avoided with:
* TFP410a (RGBTTL to DVI/HDMI)
* SN75LVDS83b (RGBTTL to LVDS)
* SSD2828 (RGBTTL to MIPI)
* various other converter ICs
4k btw is MENTAL bandwidth. multiply 3840 by 2160, then by 4, then by 60: this gives the number of bytes per second required of the internal memory bus.
turns out to be 2 gigabytes per second, doesn't it?
now check the datasheets on what affordable FPGA boards can do, what memory ICs they have, and whether they can cope with that level of bandwidth.
you'll find that there aren't any.
i do find it ironic that "incremental steps" are recommended, "to get something running", but lack of knowledge of the difficulty surrounding DRM and in ramping up to high speed leads to "disbelief". ah well :)
As far as memory bandwidth goes for 4K, HDR10 content is about 2 GB/s, regular SDR content is 1.5 GB/s for normal 24 bit color, and if you're running SDR content with chroma subsampling (4:2:0), you're at 750 MB/s.
If we're considering the "affordable" tier of FPGA boards:
An Artix 7 can be found on boards <= $200, and the GTP channels are capable of HDMI 2.0 speeds (6 Gb/s per lane). HDMI uses an 8/10b encoding, so the gearbox setup would be 600 MHz, which is doable on the -3 and -2/-2L speed tiers. You can also use 64 bit DDR3 @ 1066 MT/s (8.5 GB/s) on -3 speed grades and 64 bit DDR3 @ 800 MT/s on -2/-2L speed grades (6.4 GB/s).
You don't necessarily have to be rendering a 4K stream from RAM to want a 4K output. You could be generating the higher resolution content on demand and not be putting that much pressure on the memory. Think tiled graphics, or native resolution overlays over an upscaled input stream.
all the graphics software will write at that full 32BPP rate: it is unfortunately not reasonable to expect all Graphics Software to perform data compression.
yes there will be a few opportunities for compressed streaming, and funnily enough we were just thinking "what if the GPU were to do the same trick as the Amiga used to do, 40 years ago, by following the scan-lines?"
in the case of video playback you might reasonably expect that an area of the screen is "reserved" (not written to) but that when the Video-Output HDL gets to that pixel instead reads directly from a completely different device: one that is formatted already in HDR/SDR or YUV.
effectively this is a modern "sprite" engine.
... but, again, we have significantly diverged from the original "Just Make It Real Simple Why Don't You Just", yeh? :)
Is this true, and is it of relevance here?
IIRC most companies out there selling GPU IP are still paying royalties to AMD for their patents on shader architecture which they got from their acquisition of ATI which in turn came from their acquisition of ArtX which was founded by people who worked at the long defunct SGI (Silicon Graphics).
The funny thing is, if you backtrack through all GPU innovations, most stem from former SGI employees.
When 3Dfx went under, even though Nvidia's GPU tech was already superior to anything 3Dfx had, Nvidia immediately swept in and picked their carcass clean, mostly for their patents in this space, so they would have more ammo/leverage against competitors going forward.
Regardless how you feel about patents, with their pros and cons, hardware engineering is a capital intensive business and without patents to protect your expensive R&D, it wouldn't be a viable business.
If someone were to clean-room design their own GPU chip, how likely is it that Nvidia and AMD would come down on them anyway simply by virtue of the fact that they (presumably) have patents on everything that you could think of putting in that chip?
In essence: do you now have to be an expert in what you’re not allowed to put in before you even start?
If you want to design a GPU IP that works around all their patents, you probably can, but unless you're a John Carmack x10, your resulting design would be horribly inefficient and not competitive enough to be worth the expensive silicon it will be etched on and probably not compatible to any modern API like Vulcan or DirectX.
But if you just want to build your own meme GPU for education/shits and giggles, that doesn't follow any patents or APIs, then you can and some people already did:
I think the chance you would be sued is near 100%. If you released and showed any market traction at all, you would immediately become a threat to the duopoly; they surely remember the rise of 3Dfx. Don’t bother arguing the merits of the patents because it would be a business decision, not a technical one—this is the kind of thing that’s decided at the C-level and then justified (or cautioned against) by the company’s legal team, not the other way around. Patents are merely leverage to effect the defense of the business, and you can be sure they’ll be used.
Then, if you wanted to run a traditional rasterization pipeline, you'd do it basically in software, using approaches similar to cudaraster (which is BSD licensed!). The paper on that suggests that it would be on the order 2X slower than optimized GPU hardware for triangle-centric workloads, but that might be worth it. The good news is this story gets better the more the workload diverges from what traditional GPUs are tuned for - in particular, the more sophisticated the shaders get, the more performance depends on the ability to just evaluate the shader code efficiently.
It would of course be very difficult to make a chip that is competitive with modern GPUs (the engineering involved is impressive by any standards), but I think a lot would be gained from such an effort.
I should probably disclaim that this is definitely not legal advice. Anyone who wants to actually play in the GPU space should plan on spending some quality time with a team of topnotch lawyers.
have you seen Tom Forsyth's fascinating and funny talk about how Larrabee turned into AVX512 after 15 years?
https://player.vimeo.com/video/450406346 https://news.ycombinator.com/item?id=15993848
if it were done. say, as a Libre/Open processor, say, with the backing of NLnet (a Charitable Foundation), where NLnet has access to over 450 Law Professors more than willing to protect "Libre/Open" projects from patent trolls by running crowd-funded patent-busting efforts
if it were done as a Libre/Open Hybrid Processor, based on extending an ISA such as ooo, I dunno, maybe OpenPOWER, which has the backing of IBM with a patent portfolio spanning several decades, who would be very upset if tiny companies like NVidia or AMD tried it on against a Charitably-funded project.
that would be a very interesting situation, wouldn't it? i wonder if there's a project around that's trying this as a strategy? hmmm, hey, you know what? there is! it's called http://libre-soc.org
you are absolutely correct in that everything has moved on from "Fixed Function" of SGI, and how GPLGPU works (worked) - btw it's NOT GPL-licensed: Frank sadly made his own license, "GPL words but with non-commercial tacked onto the end" which ... er... isn't GPL... sigh - but everything commercially has now moved on to Shader Engines.
that basically means Vulkan.
however you may be fascinated to know, from Jeff's evaluation, that there are still startling similarities in basic functionality in not-GPL GPLGPU and in modern designs targetted at Shader Engines.
It isn't those patents specifically, but the IP developed on top of the safe harbor that those additional patents provided at the time.
It is difficult at any particular point in time to develop IP in this space without infringing patents that are still in force, because even if you go back and base your work on patents that have expired, unless you are very careful and clever you will be infringing on newer patents that are themselves also based on those expired patents.
You may be able to show that some of the key claims of those newer patents were "obvious to a person having ordinary skill in the art" and/or come up with prior art to give yourself some wiggle room, but that's a lot of effort with an uncertain result.
A surer strategy is to patent new developments on top of current IP that, while you can't use them without licensing the original IP, neither can the holder of the original IP, and if your new stuff is in their critical path and engineering around it would be annoying enough, you may be able to get them to negotiate a cross-licensing arrangement or something similar.
IOW, if someone's moat is stopping you or their tolls are too high, start digging a moat around their moat to get them to cut you a deal.
It's still tricky though: you can't patent stuff too far ahead of where the original IP holder is going because you may guess wrong about where they (or the industry) is going to have to go, or even if you're right your patents may expire before they become important (and you ran out of money to keep building more IP on top because your innovation wasn't implementable yet or you were too early to market), and if you aren't far enough ahead you run the risk of them patenting whatever innovation you are working on before you do, or even having the "obvious to a person with ordinary skill in the art" shoe on the other foot (and their legal department is bigger than yours).
That's surely not a healthy situation either. Courts should never be a central part of competition among businesses.
Isn't that the idea, you have a patent for 10-20 years, build your business (which AMD/nVidia did, very successfully) then everyone is free to use it, possibly leading to innovation?
I'm poorly versed in this, so if anyone with more knowledge could share some thoughts, that would be appreciated.
The original patent still expires. The problem is when the patented improvement is obvious enough that anyone who wants to build on the expired patent is going to want to do it the way the new patent does it (ie. patent N+1 is "just" a modernized reimplementation of patent N), but un-obvious enough (to a "person of ordinary skill in the art", superficially at least) that it is still patentable.
Alternatively, the original patent holder throws money and people at the problem and patents every variation on their original patent they can conceive of, and every N*M combination with their other patents, even ones they have no intention of reducing to practice, maybe even ones that seem nonsensical, just in case. IBM used to be notorious for this.
Technology moves so fast that perhaps tech patents should be given shorter terms. As it stands, the big companies just build up huge patent portfolios which discourage competition.
Any implementation of it has the exact same patent minefield to navigate as any other ISA. Most of the patents are around implementation techniques not instruction set.
https://riscv.org/technical/specifications/risc-v-genealogy/
Of course, the various extensions might violate current patents (I would guess that packed SIMD and cryptography extensions are particularly at risk). But the basic ISA does not use anything that was not already widely adopted by 2003.
Besides that, engineers in large companies are told to explicitly not look up any patents so they won't be accused of willful infringement.
That can work, but only up to the point that a disgruntled former employee blows the whistle.
Ben Eater - Let’s build a video card!
Embedded Thoughts Blog - Driving a VGA Monitor Using an FPGA
https://embeddedthoughts.com/2016/07/29/driving-a-vga-monito...
Ken Shirriff - Using an FPGA to generate raw VGA video:FizzBuzz with animation
http://www.righto.com/2018/04/fizzbuzz-hard-way-generating-v...
Clifford Wolf - SimpleVOut -- A Simple FPGA Core for Creating VGA/DVI/HDMI/OpenLDI Signals
https://github.com/cliffordwolf/SimpleVOut
PDS: Also, this looks interesting, from SimpleVOut:
>"svo_vdma.v
A video DMA controller. Has a read-only AXI4 master interface to access the video memory."
"World’s First 1k-Processor Chip, Powered by a Single AA Battery (2016)"
https://news.ycombinator.com/item?id=26140957
...Might make a good candidate for a future GPU!
...As might any massively-parallel "CPU" architecture, for example, the Adapteva Parallella:
https://en.wikipedia.org/wiki/Adapteva
>"The 16-core Parallella has roughly 5.0 GFLOPs/W, and the 64-core Epiphany-IV made with 28 nm estimated as 50 GFLOPs/W (single-precision),[27] and 32-board system based on them has 15 GFLOPS/W.[28] For comparison, top GPUs from AMD and Nvidia reached 10 GFLOPs/W for single-precision in 2009–2011 timeframe.[29]"
In other words, whenever you hear about a CPU with a "massive amount of cores" -- think "could make a great GPU", or "could make a great GPU with the right adjustments" -- from this point forward!
In other words, any massively parallel multi-core CPU architecture -- becomes or can become -- a candidate architecture to become a GPU -- with further changes/optimizations...
Before doing that, though, they should research the difference between SIMD and MIMD. Multi-core CPUs are generally MIMD, and GPUs are always SIMD†.
† If someone knows of a MIMD GPU architecture, please share!
still haven't found a consumer graphics card with a MIMD architecture. I'm guessing there might be more on-going research in this area with the GPGPU on FPGA communities.
If you're playing with FPGA's, you might as well directly compile the kernel into a circuit, rather than building a gpu on an FPGA and running your kernel on that.
Proprietary solutions like Altera OpenCL compiler exist.
Love GPGPU, I git clone it and try to understand the code better here:
https://www.code-scope.com/s/s/u#c=sd&uh=0f2c2fa280a2&h=afe7a329&di=-1&i=38
Are you writing your own kernel from scratch? If so which CPU does it runs on - some embedded CPU inside FPGA?
In the mandelbrot.c code, it has following: #define vector_mixi __builtin_nyuzi_vector_mixi
How does it get translate to vector operations in FPGA? Where is the code implement the __builtin_*?
This new iGPU has similar performance with a discrete Nvidia's GPU two years back, GeForce GT 1030!
[1]https://www.notebookcheck.net/Intel-s-Elkhart-Lake-SoC-will-...
Does anyone have good references on extending cmake to new tools that don’t produce executables per se, or otherwise work in non traditional ways?
FPGAs typically do contain dedicated RAM areas, because implementing it out of FPGA logic slices is terribly inefficient.
(* ram_style = "distributed" *) reg [data_size-1:0] myram [2**addr_size-1:0];
block: Instructs the tool to infer RAMB type components.
distributed: Instructs the tool to infer the LUT RAMs.
registers: Instructs the tool to infer registers instead of RAMs.
ultra: Instructs the tool to use the UltraScale+TM URAM primitives.
See: https://www.xilinx.com/support/documentation/sw_manuals/xili...
edit: formatting*
Finding a mentor is very difficult for me unfortunately (from a third world country, and not particularly rich). I will check out the books and research if there are any obsoleted / old parts. Thanks for the heads-up on SystemVerilog.
Thanks again :)
Instead of going with the much more expensive SRAM, someone decided that refreshing billions of capacitors hundreds of times a second while performing read and write operations is an acceptable way of storing data (even if only while powered).
I wonder what the managers who first heard the idea must've thought :D
And it works so well! It's probably one of the most reliable component in a computer.
Appears to have been invented by Dennard of Dennard Scaling: https://www.thoughtco.com/who-invented-the-intel-1103-dram-c...
GPU = CPU, but one that is placed specifically on the graphics card, and one that may specifically be optimized to perform instructions related to 3D calculations faster than other instructions.
As far as the 3D stuff, don't think "3D", instead think Math specifically related to 3D.
OK, so what is that Math?
Well, we could consult Stack Overflow for that:
https://stackoverflow.com/questions/1320403/math-used-in-3d-...
OK, so next question, what does a GPU do then?
Well, remember that a CPU can run any generalized algorithmic computations -- including those required for 3D graphics(!) (they are Turing-Complete, after all!), but the thing is, 3D graphics cards have their GPU's, er, CPU's, er, GPU's -- specifically optimized in various different ways (parallelism, specific instruction types, etc.) for the mathematics of 3D.
So you see, what is being searched for is as follows:
The fastest CPU, er, GPU, er CPU, that can be created on an FPGA, that can be optimized for specific 3D MATH operations, that can be created patent-free.
Now, here's the thing...
In Mathematics, TMTOWTDI (There's more than one way to do it) -- is King.
While some math algorithms (side philosophical question to self, "How does anyone patent MATH?". No really, how do they do it? That idea is utterly mind boggling and self-contradictory! It's like what the Principal said to Billy Madison: "Mr. Madison, what you've just said is one of the most insanely idiotic things I have ever heard. At no point in your rambling, incoherent response were you even close to anything that could be considered a rational thought. Everyone in this room is now dumber for having listened to it. I award you no points, and may God have mercy on your soul."), while some math algorithms implemented in hardware are patented (and possibly still under patent) -- there are many others that are not!
In other words, there are other ways to get the job done...
Some good starting points might be to read up on the 3D hardware in the original Playstation (1994) https://en.wikipedia.org/wiki/PlayStation_technical_specific...
>"Geometry Transformation Engine (GTE)
Coprocessor [PDS: aka "GPU"] that resides inside the main CPU processor, giving it additional vector math instructions used for 3D graphics, lighting, geometry, polygon and coordinate transformations – GTE performs high-speed matrix multiplications.
Well, there we go, there's the start of the 3D math part, vector math instructions and high-speed matrix multiplications.
Then there is "Fused Multiply Add", aka "Multiply Accumulate Operation":
https://en.wikipedia.org/wiki/Multiply%E2%80%93accumulate_op...
>"A fast FMA can speed up and improve the accuracy of many computations that involve the accumulation of products:
Dot product Matrix multiplication Polynomial evaluation"
PDS: Which would be used in 3D Graphics acceleration...
Anyway, those are the starting points to look into if someone wants to "leap off" into this area...
This is an interesting topic; it sort of exists as confluence of several disciplines -- including Digital Logic Engineering, Algorithms, Law and Math...
(But seriously, how does anyone patent MATH itself? I mean, seriously! <g>)