"Processor-in-Memory", is an age-old idea which was even explored in a doctoral thesis from 1998: "Computational RAM: A Memory-SIMD Hybrid"*. At the time, the motivation was, "Between a DRAM chip and a microprocessor mounted on a circuit board, we can only get a few hundred megabytes per second of bandwidth. However, inside the memory chip itself, there's hundreds of gigabytes (terabits) of bandwidth available, so why not move some of the logic there?". And then the author pushed it to the extreme where there was a wide array of 1-bit processors at each column of DRAM. (Note: the reason why true PIM isn't viable is because of the radically different semiconductor process used for DRAM vs. high-speed digital logic, i.e.: your digital logic circuits will perform poorly on a "DRAM process").
The designers of HBM5 had the same observations and thought, "why don't we put a logic chip at the bottom of the stack of HBM DRAM chips and get extremely high memory bandwidth?" But, in looking at this, you'll also quickly realize this is almost exactly what Intel and Micron pitched 15 years ago as Hybrid Memory Cube** (which lost on the market compared to the simpler, less costly, and more flexible alternative, HBM)
Duncan Elliott taught my graduate (IIRC) computer architecture course while writing that thesis - we all talked about C-RAM for a while.. And I think it was a rehash of an even older paper / discussion about that idea.
Compute In Memory has been most of what electronics research (more accurately, the parts I am interacting with) talking about this year. There are a few trends and I am curious to see which ones survive in the market. RAM designers are hot in the market and job openings at the moment.
I like the idea of a CPU/GPU/TPU on RAM (look at those gorgeous bandwidth numbers!) -- but I also wonder what problems it may potentially bring in the future... I think it would be a good idea that if RAM is to implement processors / PIM on it in the future, that each RAM stick should have an external DIP "safety" switch on the side which switches off such processing. If this switch is flipped by the user, then the stick simply falls back to operating as normal "dumb" RAM, driven by external CPU control. Also, it would be a good idea to have debug ports, debug proxying, single step capability, etc., in these futuristic RAM sticks, as well...
Related:
"SIMDRAM: A Framework for Bit-Serial SIMD Processing Using DRAM Extended Abstract" (2020):
>"We leverage the SIMDRAM framework to accelerate seven application kernels from machine learning, databases, and image processing (VGG-13 [63], VGG-16 [63], LeNET [38], kNN [42], TPC-H [66], BitWeaving [45], Brightness [28]). SIMDRAM provides up to 2.5× speedup for the kernels compared to Ambit [58]. Compared to a CPU and a high-end GPU, SIMDRAM is 257× and 31× more energy efficient, while providing 93 × and 6× higher throughput, respectively."
You seem to think that "compute in memory" would mean the disappearance of commodity DRAM parts and their replacement with parts that had compute capabilities. This is impossible. Adding compute to memory will make it more expensive, so it will get zero adoption outside of the niches where it actually proves to be useful and valuable. Nobody is going to be putting compute in memory chips onto standard consumer DIMMs.
AI inference is basically 1 operation iterated over every single model weight. That's what's makes computer in memory attractive and feasible.
General purpose workloads would require much more silicon real estate and are not (typically) nearly as bandwidth hungry. Just like not all CPU workloads benefit from being adapted to a GPU, even fewer will benefit from being adapted to processing in RAM.
AI is not intrinsically matmul. Current models are a framing of the problem as matmul to enable the bulk operations necessary. There are many calculations that can (and have been) reframed into matrix form to achieve similar results. More than once I have seen a matmul version of something that I thought couldn't have been representable that way. I'm not sure if the greater benefit is the performance or the ability to conceptually manage the problem in you head.
Not all algorithms can be converted of course, but a lot of that is because there are some inherently unparallelizable problems.
Many algorithms can be reasonably converted into matmul. Not many algorithms can be reasonably converted into very low precision matmul. And that's all that these things are going to do.
I looked at this for a client close to 20 years ago. The problem is that you need to get software support for it. Arguably easier these days with open source but still doesn't magically happen.
The nice thing is that for GPUs you're rarely writing directly against the metal but instead pseudocode lowered by the driver; the driver could know to just lower it this way. On an NVidia GPU they control the whole stack and don't need to get a further ecosystem going there. Possibly the same for Apple.
I wonder if we might eventually see stacking of NAND, logic, and DRAM dies together so each can still be fabbed on their specialized process. Getting all three on one wafer is extremely impractical, but having them all connected with TSVs or other bonding might be feasible, if the density of the respective dies works out right to give you the right ratio of storage for weights, working memory, and compute.
"Processor-in-Memory", is an age-old idea which was even explored in a doctoral thesis from 1998: "Computational RAM: A Memory-SIMD Hybrid"*. At the time, the motivation was, "Between a DRAM chip and a microprocessor mounted on a circuit board, we can only get a few hundred megabytes per second of bandwidth. However, inside the memory chip itself, there's hundreds of gigabytes (terabits) of bandwidth available, so why not move some of the logic there?". And then the author pushed it to the extreme where there was a wide array of 1-bit processors at each column of DRAM. (Note: the reason why true PIM isn't viable is because of the radically different semiconductor process used for DRAM vs. high-speed digital logic, i.e.: your digital logic circuits will perform poorly on a "DRAM process").
The designers of HBM5 had the same observations and thought, "why don't we put a logic chip at the bottom of the stack of HBM DRAM chips and get extremely high memory bandwidth?" But, in looking at this, you'll also quickly realize this is almost exactly what Intel and Micron pitched 15 years ago as Hybrid Memory Cube** (which lost on the market compared to the simpler, less costly, and more flexible alternative, HBM)
* https://www.eecg.toronto.edu/~stumm/Theses/Elliott-PhD98.pdf
** https://en.wikipedia.org/wiki/Hybrid_Memory_Cube
Age-old... 1998... :grimace:
Duncan Elliott taught my graduate (IIRC) computer architecture course while writing that thesis - we all talked about C-RAM for a while.. And I think it was a rehash of an even older paper / discussion about that idea.
Compute In Memory has been most of what electronics research (more accurately, the parts I am interacting with) talking about this year. There are a few trends and I am curious to see which ones survive in the market. RAM designers are hot in the market and job openings at the moment.
I like the idea of a CPU/GPU/TPU on RAM (look at those gorgeous bandwidth numbers!) -- but I also wonder what problems it may potentially bring in the future... I think it would be a good idea that if RAM is to implement processors / PIM on it in the future, that each RAM stick should have an external DIP "safety" switch on the side which switches off such processing. If this switch is flipped by the user, then the stick simply falls back to operating as normal "dumb" RAM, driven by external CPU control. Also, it would be a good idea to have debug ports, debug proxying, single step capability, etc., in these futuristic RAM sticks, as well...
Related:
"SIMDRAM: A Framework for Bit-Serial SIMD Processing Using DRAM Extended Abstract" (2020):
https://arxiv.org/html/2012.11890v1
>"We leverage the SIMDRAM framework to accelerate seven application kernels from machine learning, databases, and image processing (VGG-13 [63], VGG-16 [63], LeNET [38], kNN [42], TPC-H [66], BitWeaving [45], Brightness [28]). SIMDRAM provides up to 2.5× speedup for the kernels compared to Ambit [58]. Compared to a CPU and a high-end GPU, SIMDRAM is 257× and 31× more energy efficient, while providing 93 × and 6× higher throughput, respectively."
You seem to think that "compute in memory" would mean the disappearance of commodity DRAM parts and their replacement with parts that had compute capabilities. This is impossible. Adding compute to memory will make it more expensive, so it will get zero adoption outside of the niches where it actually proves to be useful and valuable. Nobody is going to be putting compute in memory chips onto standard consumer DIMMs.
"DRAM banks already provide most of the bandwidth; the external pins cannot expose it." Closed the tab.
Why would you want to read something someone was so sloppy at they didn't improve it with AI?
AI focussed - as so much these days. Some corp. please put general-purpose cores in that RAM?
AI inference is basically 1 operation iterated over every single model weight. That's what's makes computer in memory attractive and feasible.
General purpose workloads would require much more silicon real estate and are not (typically) nearly as bandwidth hungry. Just like not all CPU workloads benefit from being adapted to a GPU, even fewer will benefit from being adapted to processing in RAM.
AI is not intrinsically matmul. Current models are a framing of the problem as matmul to enable the bulk operations necessary. There are many calculations that can (and have been) reframed into matrix form to achieve similar results. More than once I have seen a matmul version of something that I thought couldn't have been representable that way. I'm not sure if the greater benefit is the performance or the ability to conceptually manage the problem in you head.
Not all algorithms can be converted of course, but a lot of that is because there are some inherently unparallelizable problems.
Many algorithms can be reasonably converted into matmul. Not many algorithms can be reasonably converted into very low precision matmul. And that's all that these things are going to do.
I looked at this for a client close to 20 years ago. The problem is that you need to get software support for it. Arguably easier these days with open source but still doesn't magically happen.
The nice thing is that for GPUs you're rarely writing directly against the metal but instead pseudocode lowered by the driver; the driver could know to just lower it this way. On an NVidia GPU they control the whole stack and don't need to get a further ecosystem going there. Possibly the same for Apple.
Integrated RAM with tensor processing seems like the end game for silicon that's actually efficient at running AI models.
Or integrated flash with TPUs? That would be awesome. Flash your model onto a model-processor and run it.
I wonder if we might eventually see stacking of NAND, logic, and DRAM dies together so each can still be fabbed on their specialized process. Getting all three on one wafer is extremely impractical, but having them all connected with TSVs or other bonding might be feasible, if the density of the respective dies works out right to give you the right ratio of storage for weights, working memory, and compute.
So, long samsung and short nvidia