Nice breakdown. I've been wondering why it's so complicated to program on the GPU compared to the CPU. Is it a language paradigm issue or maybe a hardware design limitation? I always think of this when looking at the graphic APIs, espeically Vulkan.
It's mostly hardware design limitations. GPU hardware is way more diverse than CPU hardware, so you need a portable intermediate representation and a runtime shader compiler, and you need to abstract over wildly different behavior in many places, especially (but not exclusively) in older hardware.
If you want to see what a minimal API design for modern GPU hardware would look like if you dropped support for a bunch of complexity required only by older hardware, look no further than: https://github.com/sebbbi/NoGraphicsAPI and the accompanying blog post.
part of the reason is historical cruft & history path dependency. GPUs didn't start as general computing device for parallelism but a device for the specific purpose of accelerating graphics with very specific fixed functions for that job and only that. with time that fixed pipeline became more complex and more configurable, not fixed/more flexible and hacks were in parallel used to make use of it for tasks outside graphics those hacks were later made into something more formal (CUDA/OpenCL).
CUDA being the winner (closed source, not really usable on non-nvidia GPUs) between those 2 and nvidia not wanting to open their competitive advantage to the competition for the niche of non gamings uses / scientific computing (and later AI, which became a huge win for nvidia).
vulkan came around after as this open thing (based on a previous attempt by AMD called mantle) that's trying to break down the leaky abstractions and the divide between language for graphics & video games and separate language/frameworks for gpu non-graphic computing with the promise of giving more control and more room for squeezing performance from the hardware at the cost of more complexity with the idea that you'll probably build your own abstractions on top of it for your subset of applications.
If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (and if the competition between the duopoly didn't sabotage things)
don't forget that the drivers for GPUs would until recently get updates to patch specific video games for performance issues (not sure if they still do that ?)
Good summary, thanks. Sounds like it's all still pretty early, even for graphics but much more for parallel compute, which seems the direction the hardware is going in. I'm sure there is a CUDA equivalent for CPU but I would guess it's all the way back in the 70s or earlier. Hopefully open standards gets some traction.
> If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (...)
The market demand was for video game tools, though.
People didn't wanted faster compute, they wanted their computer games to work smoothly on the hardware they already own. Therefore, they preferred to spend 100-200€ on a point upgrade, much like RAM, than scrapping their 2k€ PC.
To give people a reason to buy these upgrades, games had to use them. Therefore video cards were designed specially for the usecases and development flow that game developers already followed.
If it wasn't for the video game market, NVidia wouldn't exist as we know it today.
It's not more complicated, you have things like CUDA which is basically C/C++ that compiles to GPU.
But GPUs have thousands of cores, which means if you want any benefit, the smallest data type you must use is something like array[n * 1024], or your GPU is wasted doing nothing. So you must shape your programs into that memory model or you're just wasting your time (and GPU).
Right, that makes sense. There's a ton of overhead in the setup though, I'm assuming that's because everything runs on the CPU by default. Is this then an OS design choice? Thanks for your input.
There can be overhead, though GPU memory + drivers are some of the most optimized software in the world.
Some OS/hardware have separate GPU memory (Nvidia), some have unified memory (Macos, modern consoles), but moving memory is usually not the problem.
Things don't "run on CPU by default", it's up to the programmer where to run it. But the CPU and GPU paradigm of "running a program" are very different, since on GPU you're running SIMD (i.e. running your code on arrays) or else there is no point to use a GPU.
For me, if-branching is the hardest to wrap my head around. "If"s are so natural to CPU programming that it's second nature to any developer but on GPU, it's expensive.
It's partially historical; I'm sure if we redesigned GPGPU today from the ground up, we'd have an entirely different pipeline. And partially due to hardware vendor lock-in; you can't really design an alternative to Vulkan/d3d12 even if you had the skill and drive to do so.
But also partly in the fact that the GPU is specialized for certain operations over a CPU. It wants and need to obtain data in a different way from a CPU to maximize throughput. So language constructs are designed around making sure developers stay on this fast path. The complexity can be seen as some intentional gatekeeping to signal that this kind of programming is only for specific kinds of problems.
GLES isn't a bad way to teach graphics but I wish there was a middle ground between it and VK.
The worst part of GL is the mysterious global state you will never understand without studying the spec. The world could use an API that's basically GL but without global hidden singleton (per thread!) BS behind the scenes.
I've learned more graphics writing software rasterizers and ray tracers than using any APIs. If the goal is to teach GPU rendering specifically, then OpenGL is indeed more amenable to teaching than the others. But if the goal is to teach graphics and rendering concepts, then OpenGL et al are just miserable and counter-productive.
Disagree.
Trying to express all of graphics and rendering through the lens of OpenGL will of course fail and doing 3D rendering itself via raytracing is simply not possible in rasterization focused OpenGL without dirty workarounds. That is obvious.
A fundamental shift in thinking that got me into high performance graphics programming and parallelized image processing, the "Aha" Moment, is being forced to express how to calculate the final pixel independently, without having access to neighboring pixels of the same output buffer, confined to the straight jacket of the fragment shader. OpenGL with it's combination of hand-holding and relative simplicity as compared to Vulkan and limits of a predefined shader pipeline as compared to a pure software implementation is exactly the thing that I needed to get into graphics programming. It's an excellent jump in point I think.
Nice breakdown. I've been wondering why it's so complicated to program on the GPU compared to the CPU. Is it a language paradigm issue or maybe a hardware design limitation? I always think of this when looking at the graphic APIs, espeically Vulkan.
It's mostly hardware design limitations. GPU hardware is way more diverse than CPU hardware, so you need a portable intermediate representation and a runtime shader compiler, and you need to abstract over wildly different behavior in many places, especially (but not exclusively) in older hardware.
If you want to see what a minimal API design for modern GPU hardware would look like if you dropped support for a bunch of complexity required only by older hardware, look no further than: https://github.com/sebbbi/NoGraphicsAPI and the accompanying blog post.
part of the reason is historical cruft & history path dependency. GPUs didn't start as general computing device for parallelism but a device for the specific purpose of accelerating graphics with very specific fixed functions for that job and only that. with time that fixed pipeline became more complex and more configurable, not fixed/more flexible and hacks were in parallel used to make use of it for tasks outside graphics those hacks were later made into something more formal (CUDA/OpenCL).
CUDA being the winner (closed source, not really usable on non-nvidia GPUs) between those 2 and nvidia not wanting to open their competitive advantage to the competition for the niche of non gamings uses / scientific computing (and later AI, which became a huge win for nvidia).
vulkan came around after as this open thing (based on a previous attempt by AMD called mantle) that's trying to break down the leaky abstractions and the divide between language for graphics & video games and separate language/frameworks for gpu non-graphic computing with the promise of giving more control and more room for squeezing performance from the hardware at the cost of more complexity with the idea that you'll probably build your own abstractions on top of it for your subset of applications.
If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (and if the competition between the duopoly didn't sabotage things)
don't forget that the drivers for GPUs would until recently get updates to patch specific video games for performance issues (not sure if they still do that ?)
Good summary, thanks. Sounds like it's all still pretty early, even for graphics but much more for parallel compute, which seems the direction the hardware is going in. I'm sure there is a CUDA equivalent for CPU but I would guess it's all the way back in the 70s or earlier. Hopefully open standards gets some traction.
> If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (...)
The market demand was for video game tools, though.
People didn't wanted faster compute, they wanted their computer games to work smoothly on the hardware they already own. Therefore, they preferred to spend 100-200€ on a point upgrade, much like RAM, than scrapping their 2k€ PC.
To give people a reason to buy these upgrades, games had to use them. Therefore video cards were designed specially for the usecases and development flow that game developers already followed.
If it wasn't for the video game market, NVidia wouldn't exist as we know it today.
It's not more complicated, you have things like CUDA which is basically C/C++ that compiles to GPU.
But GPUs have thousands of cores, which means if you want any benefit, the smallest data type you must use is something like array[n * 1024], or your GPU is wasted doing nothing. So you must shape your programs into that memory model or you're just wasting your time (and GPU).
That's >50% of the problem.
Right, that makes sense. There's a ton of overhead in the setup though, I'm assuming that's because everything runs on the CPU by default. Is this then an OS design choice? Thanks for your input.
There can be overhead, though GPU memory + drivers are some of the most optimized software in the world.
Some OS/hardware have separate GPU memory (Nvidia), some have unified memory (Macos, modern consoles), but moving memory is usually not the problem.
Things don't "run on CPU by default", it's up to the programmer where to run it. But the CPU and GPU paradigm of "running a program" are very different, since on GPU you're running SIMD (i.e. running your code on arrays) or else there is no point to use a GPU.
For me, if-branching is the hardest to wrap my head around. "If"s are so natural to CPU programming that it's second nature to any developer but on GPU, it's expensive.
It's partially historical; I'm sure if we redesigned GPGPU today from the ground up, we'd have an entirely different pipeline. And partially due to hardware vendor lock-in; you can't really design an alternative to Vulkan/d3d12 even if you had the skill and drive to do so.
But also partly in the fact that the GPU is specialized for certain operations over a CPU. It wants and need to obtain data in a different way from a CPU to maximize throughput. So language constructs are designed around making sure developers stay on this fast path. The complexity can be seen as some intentional gatekeeping to signal that this kind of programming is only for specific kinds of problems.
GLES isn't a bad way to teach graphics but I wish there was a middle ground between it and VK.
The worst part of GL is the mysterious global state you will never understand without studying the spec. The world could use an API that's basically GL but without global hidden singleton (per thread!) BS behind the scenes.
That middle ground is often called DirectX 11 but sadly it's stuck to a specific platform.
Sounds like a great way to make sure students understand fundamental concepts.
The linked webpage refuses keyboard navigation e.g. cursors, pageup/down and home/end seem to do nothing.
Awful.
I think it's because of the webgl demo at top stealing focus
Specifically, it registers a keydown handler on the <html> that swallows everything.
I've learned more graphics writing software rasterizers and ray tracers than using any APIs. If the goal is to teach GPU rendering specifically, then OpenGL is indeed more amenable to teaching than the others. But if the goal is to teach graphics and rendering concepts, then OpenGL et al are just miserable and counter-productive.
Disagree. Trying to express all of graphics and rendering through the lens of OpenGL will of course fail and doing 3D rendering itself via raytracing is simply not possible in rasterization focused OpenGL without dirty workarounds. That is obvious.
A fundamental shift in thinking that got me into high performance graphics programming and parallelized image processing, the "Aha" Moment, is being forced to express how to calculate the final pixel independently, without having access to neighboring pixels of the same output buffer, confined to the straight jacket of the fragment shader. OpenGL with it's combination of hand-holding and relative simplicity as compared to Vulkan and limits of a predefined shader pipeline as compared to a pure software implementation is exactly the thing that I needed to get into graphics programming. It's an excellent jump in point I think.
The way I’m interpreting their post is, you could far more easily teach things like introductory projective geometry without a GPU.