A good technical project, but honestly useless in like 90% of scenarios.
You want to use an NVidia GPU for LLM ? just buy a basic PC on second hand (the GPU is the primary cost anyway), you want to use Mac for good amount of VRAM ? Buy a Mac.
With this proposed solution you have an half-backed system, the GPU is limited by the Thunderbolt port and you don’t have access to all of NVidia tool and library, and on other hand you have a system who doesn’t have the integration of native solution like MLX and a risk of breakage in future macOS update.
Evidence that NVIDIA has even been trying? My understanding is that Apple didn’t allow 3rd parties to write graphics drivers past 10.13, but they could’ve done a non-graphics driver like this.
The government doesn’t care? They’re a minority of the market? The vast majority of their computers didn’t have slots to put Nvidia GPUs in, and now none of them do?
It depends how you define the market. In the 2001 microsoft case [0], the courts ruled Microsoft had a monopoly over the "Intel-based personal computer market".
Apple has a monopoly over the "M-chip" personal computer market. They have a monopoly over the iOS market with the app store. They have a monopoly over the driver market on macOS.
Like, Microsoft was found guilty of exploiting its monopoly for installing IE by default while still allowing other browser engines. On iOS, apple bundles safari by default and doesn't allow other browser engines.
If we apply the same standard that found MS a monopoly in the past, then Apple is obviously a monopoly, so at the very least I think it's fair to say that reasonable people can disagree about whether Apple is a monopoly or not.
I wouldn’t say it is obvious. Apple does not have the monopoly of ARM based PCs. Labeling it as a monopoly of M chips is not fair or accurate when comparing to MS on Intel. It’s also probably relevant that MS was not selling PCs or their own hardware. They had a monopoly on a market where you effectively had to use their software to use the hardware you bought from a different company. Because Apple is selling their own hardware and software as a single product, the consumer is not forced into restricting the hardware they bought by a second company’s policies.
> Apple has a monopoly over the "M-chip" personal computer market. They have a monopoly over the iOS market with the app store
When a company is deemed an illegal monopoly, the DoJ basically becomes part of management. Antitrust settlements focus on germane elements, e.g. spin offs. But they also frequently include random terms of political convenience.
I don’t think we want a precedent where companies having a product means they have an automatic monopoly on said product.
I don't think any of what you're describing are legal "monopolies". I don't have a single Apple product in my life but I'm fairly sure there's nothing I'm prevented from doing because of that.
And back in the "Microsoft has a monopoly on IE6" ruling's days, I did not use Windows or Internet Explorer, and I was not prevented from doing anything because of that. Netscape Navigator on Linux worked fine. Sure, I occasionally hit sites that were broken and only worked in IE, but I also right now frequently hit apps that are "macOS only" (like when Claude Cowork released, or a ton of other YC company's apps).
Microsoft was found guilty, so clearly the bar is not what you're trying to claim.
Microsoft was found guilty of using their market power to do product bundling, which is illegal. The fact that they had dominance in the market is not what they got popped for, nor is it illegal.
Apple has not, to my knowledge, required OEMs to bundle Safari with macOS alongside threats to withhold macOS if they don’t comply expressly to put Firefox out of business.
But hey, maybe some weird shit happened during the clone years that I’m not privy to.
You were not prevented from doing anything, but that doesn’t mean others weren’t. For example, OEMs were not allowed to offer any other preinstalled OS as a default option. That effectively killed Be and I’m sure hindered RedHat.
Yes, but that was coupled with other factors like them strongarming vendors, already being hugely dominant on desktops and abusing that position et al. I don't see this as being the same. Maybe my bar here is wrong, but it doesn't change whether they are a monopoly or not.
That’s not how monopoly definitions work. That makes about as much sense as saying Nintendo has a monopoly on Nintendo consoles or Ford has a monopoly on Mustangs
Courts have already ruled it does in the iOS app store market. You can disagree of course but then you'd be disagreeing with legal experts who know more about anti-trust law than you do.
It isn't just about monopoly or unfair competition. This can also be covered under consumer rights - the Right to Repair. No OS provider should be allowed to dictate what software you can or not run on your own device and / or OS you have paid for.
> It isn't just about monopoly or unfair competition. This can also be covered under consumer rights - the Right to Repair.
If we have a right to repair (we broadly do not, AFAICT), then that doesn't necessarily mean that we have a right to modify and/or add new functionality.
When I repair a widget that has become broken, I merely return it to its previous non-broken state. I might also decide to upgrade it in some capacity as part of this repair process, but the act of repairing doesn't imply upgrades. At all.
> No OS provider should be allowed to dictate what software you can or not run on your own device and / or OS you have paid for.
I agree completely, but here we are anyway. We've been here for quite some time.
Internet Explorer Mobile is a YouTube client. You're describing a client-server disagreement when the user is talking about an entirely client-based conflict.
> That's normal behavior when your server is being reverse-engineered or abused. Video bandwidth is not free.
Microsoft rewrote their Windows Phone native client to pass through Google's ads. Google still blocked it.
Was it normal behavior when Google blocked Amazon Fire devices from connecting to YouTube with a web browser during the Google/Amazon corporate spat?
To be fair, Google did back down almost immediately when the tech press picked up on it.
Not allowing a native client for your monopoly market share video service on Amazon devices while also blocking Amazon's web browser on those devices is making things a bit too obvious.
I followed the instructions link and read the scripts...although the TinyGPU app is not in source form on GitHub, this looks to me like the GPU is passed into the Linux VM underneath to use the real driver and then somehow passed back out to the Mac (which might be what the TinyGrad team actually got approved).
Or I could have totally misunderstood the role of Docker in this.
My read of everything is that they are using Docker for NVIDIA GPUs for the sake of "how do you compile code to target the GPU"; for AMD they're just compiling their own LLVM with the appropriate target on macOS.
As more people carry ARM laptops and keep the GPU somewhere else, I think the interesting UX question becomes whether the GPU can "follow" the local workflow instead of forcing the whole workflow to move to the GPU host. That's the problem we've been looking at with GPUGo / TensorFusion: local-first dev flow, remote GPU access when needed. Curious whether people here mostly want true attached-eGPU semantics, or just the lowest-friction way to access remote compute from a Mac without turning everything into a remote desktop / VM workflow.
I'm writing scientific software that has components (molecular dynamics) that are much faster on GPU. I'm using CUDA only, as it's the eaisiest to code for. I'd assumed this meant no-go on ARM Macs. Does this news make that false?
Woah, this is exciting. I'm traveling but I have a 5090 lying around at home. I'm eager to give it a go. Docs are here: https://docs.tinygrad.org/tinygpu/
I hope it'll work on an M4 Mac Mini. Does anyone know what hardware to get? You'll need a full ATX PSU to supply power, right? And then tinygrad can do LLM inference on it?
I own one of these, the cage is just a piece of plastic. Anyway, I don't think 80$ is that big of a difference here. I can't really afford a 4k Nvidia GPU. Intel is my only hope.
Almost twice the price and simply more accurate info regarding price and features.
Brand is TH3P4G3. Egpu.io has decent eGPU comparisons.
I wouldn't want all that dust in my GPU fans, prefer that near my case fans. I also don't like it given I got cats and want to store/box hw. I do use the eGPU in the fuse box. If I had a larger house, I'd use a server rack.
I was recently in the market for an eGPU but for a different niche (not eGPU/eNPU/eTPU but getting a HBA via TB to connect a LTO-6 drive via SAS). I went for a Sonnet instead, very low profile and small. I also bought an Asus one. Slightly bigger, came with more fans but TB4 instead of TB3 on the Sonnet. The cages are aluminium. Those eGPU were second hand (also without warranty but quicker S&H than Chinese New Year) but came with PSU. As you also gotta buy a PSU for it which came with the eGPUs I mentioned. For me no biggie, as I got a decent PSU lying around.
Maybe I’m lacking imagination. But how will a GPU with small-ish but fast VRAM and great compute, augment a Mac with large but slow VRAM and weak compute? The interconnect isn’t powerful enough to change layers on the GPU rapidly, I guess?
> But how will a GPU with small-ish but fast VRAM and great compute, augment a Mac with large but slow VRAM and weak compute?
It would work just like a discrete GPU when doing CPU+GPU inference: you'd run a few shared layers on the discrete GPU and place the rest in unified memory. You'd want to minimize CPU/GPU transfers even more than usual, since a Thunderbolt connection only gives you equivalent throughput to PCIe 4.0 x4.
My Mini is actually the smallest model so it actually has "small but slow VRAM" (haha!) so the reason I want the GPU for are the smaller Gemmas or Qwens. Realistically, I'll probably run on an RTX 6000 Pro but this might be fun for home.
“Lying around”. I’ve got an unopened 5090 in a box that I know will suffer the same fate, so I’m sending it back. So privileged to have the money to impulse buy a 5090 and yet no time to actually do anything with it.
Doesn't Apple support the major standard device categories: NVMe, XHCI, AHCI, and such, like most operating systems do? The challenges are all for hardware that needs a vendor-specific driver instead of conforming to a standard driver interface (which doesn't always exist). Lots of those can be supported with userspace drivers, which can be supplied by third parties instead of needing to be written by Apple.
Macs and PCs are fundamentally different. Their architectures have always been distinct though the Intel Mac era has somewhat blurred the line.
Modern Mac is Macintosh descendants and by contrast PC is IBM PC descendants (their real name is technically PC-clone but because IBM PC don’t exist anymore the clone part have been scrapped).
And with Apple silicon Mac the two is again very different, for example Mac don’t use NVMe, they use just nand (their controller part is integrated in the SoC) and they don’t use UEFI or BIOS, but a combination of Boot ROM, LLB and iBoot
> Why does Apple need to make the drivers in a walled garden?
Isn't that the whole point of the walled garden, that they approve things? How could they aim and realize a walled garden without making things like that have to pass through them?
> Why does Apple need to make the drivers in a walled garden?
Because third party drivers usually are utter dogshit. That's how Apple managed to get double the battery life time even in the Intel era over comparable Windows based offerings.
Such a shame both companies are big on vanity to make great things happen. Imagine where you could run Mac hardware with nvidia on linux. It's all there, and closed walls are what's not allowing it to happen. That's what we as customers lose when we forego control of what we purchase to those that sold us the goods.
Unfortunately, Apple still won't release iMessage for Android or Linux (unlike every other messenger platform, like whatsapp, telegram, wechat, microsoft teams, etc, which are all cross-platform).
Because of that, you need an apple device around to be able to deal with iMessage users.
That is no longer true. https://bluebubbles.app/
Well… it’s not exactly no longer true, you do need an Apple VM but it doesn’t have to be the end device.
I don't understand the logic for downvotes.
We vote with our wallets.
When I could not update the Ram on my personal Dell machine I asked for a Frame.work in my new job. As my Intel based FW at work had thermal throttling problems, for my next personal purchase I got an AMD one. As Ubuntu had shady practices, I installed Fedora, as Gnome forced UX choices I did not want, I used KDE. As I wanted my machine to be even more stable I use an immutable spin.
The machine I'm using now represents my choices and matches what matters to me, and works closer to perfectly than all my machines in the past
And yes, I have worked with macs, and no, the UX and the entire tyranny in the Apple ecosystem was not something I could live with
And yes, this machine is fast, predictable, a joy to work with and is a tool I control, not a tool to control me. If something happens to it, I can order the part with the same price that goes into a new machine, and keep using my laptop
"We vote with our wallet, so don't complain" is a bad take in my opinion.
Like, for phones, I want a phone which runs Linux, has NFC support, and also has iMessage so my friend who only communicates with blue-bubbles and will never message a green-bubble will still talk to me. I also want it to have regulatory approval in the country I live in so I can legally use it to make calls.
Because apple has closed the iMessage ecosystem such that a linux phone can't use it, such a device is impossible. I cannot vote for it.
As such, I will complain about every phone I own for the foreseeable future.
Well, to be fair, the whole shebang is from a completely different company, that have their own ML library and such, so that isn't that surprising. Although I agree that some CUDA shim or similar would be a lot more interesting, still getting to the place of running inference and training with your very own library is pretty dope already.
My main thought is would this allow me to speed up prompt process for large MoE models? That is the real bottleneck for m3ultra. The tokens per second is pretty good.
tinygrad does have pretty neat support for sharding things across various devices relatively easy, that'd help. I'm guessing you'd hit the bandwidth ceiling transferring stuff back and forth though instead.
I hooked up a Radeon RX 9060 XT to my Feodra KDE laptop (Yoga Pro 7 14ASP9) using a Razer Core X Chroma (40Gbps), and the performance when using the eGPU was very similar to using the Radeon 880M built into the laptop's Ryzen 9 365 APU.
So at least with my setup, performance is not great at all.
On paper, TB4 is capable of pushing 5GB/s, which is somewhere between 4x and 8x of PCIe 3.0, while a 16x PCIe 4.0 link can do ~31.5GB/s.
Well, for starters, PCIe 5.0 x16 would do something like about 60 GB/s each way, while Thunderbolt 4 does 4 GB/s each way, TB 5 does 8 GB/s each way. If you don't actually hit the bandwidth limits, it obviously matters less. Whether you'd notice a large difference would depends heavily on the type of workload.
The opportunity cost of Apple refusing to sign Nvidia's OEM AArch64 drivers is probably reaching the trillion-dollar mark, now that Nvidia and ARM have their own server hardware.
Apple got out of the server game long before they adopted aarch64, so that's a trillion worth of server hardware they never would have sold anyway. And probably not actually a trillion.
Almost everyone including myself had MacBook Pros at my last place of work.
If Apple was in the high-end server market, I see no reason why the company I was working for would not be running macOS on Apple hardware as servers, instead of the fleet of Linux based servers they had.
From what I understand, only works with Tinygrad. Which is better than nothing but CUDA or Vulkan on pytorch isn’t going to work from this.
[1] https://docs.tinygrad.org/tinygpu/
What a headline. Why does Apple have to "approve"? Do I actually own the hardware if Apple still gets to say what I can or can't run on it?
Idk why this doesn't link to the original source instead of this proxy source: https://x.com/__tinygrad__/status/2039213719155310736
Isn't X usually the original source these days?
Because it’s on the site formally known as twitter which is full of actual Nazis.
A good technical project, but honestly useless in like 90% of scenarios.
You want to use an NVidia GPU for LLM ? just buy a basic PC on second hand (the GPU is the primary cost anyway), you want to use Mac for good amount of VRAM ? Buy a Mac.
With this proposed solution you have an half-backed system, the GPU is limited by the Thunderbolt port and you don’t have access to all of NVidia tool and library, and on other hand you have a system who doesn’t have the integration of native solution like MLX and a risk of breakage in future macOS update.
Chicken/egg. NVidia tooling is lacking surely in part because the hardware wasn’t usable on macOS until now. Now that it’s usable that might change.
I don't know how Apple has evaded regulatory scrutiny for their refusal to sign Nvidia's eGPU drivers since 2018.
Evidence that NVIDIA has even been trying? My understanding is that Apple didn’t allow 3rd parties to write graphics drivers past 10.13, but they could’ve done a non-graphics driver like this.
The government doesn’t care? They’re a minority of the market? The vast majority of their computers didn’t have slots to put Nvidia GPUs in, and now none of them do?
They said eGPU
Apple doesn’t have a monopoly in any market they are in.
It depends how you define the market. In the 2001 microsoft case [0], the courts ruled Microsoft had a monopoly over the "Intel-based personal computer market".
Apple has a monopoly over the "M-chip" personal computer market. They have a monopoly over the iOS market with the app store. They have a monopoly over the driver market on macOS.
Like, Microsoft was found guilty of exploiting its monopoly for installing IE by default while still allowing other browser engines. On iOS, apple bundles safari by default and doesn't allow other browser engines.
If we apply the same standard that found MS a monopoly in the past, then Apple is obviously a monopoly, so at the very least I think it's fair to say that reasonable people can disagree about whether Apple is a monopoly or not.
[0]: https://en.wikipedia.org/wiki/United_States_v._Microsoft_Cor....
I wouldn’t say it is obvious. Apple does not have the monopoly of ARM based PCs. Labeling it as a monopoly of M chips is not fair or accurate when comparing to MS on Intel. It’s also probably relevant that MS was not selling PCs or their own hardware. They had a monopoly on a market where you effectively had to use their software to use the hardware you bought from a different company. Because Apple is selling their own hardware and software as a single product, the consumer is not forced into restricting the hardware they bought by a second company’s policies.
> Apple has a monopoly over the "M-chip" personal computer market. They have a monopoly over the iOS market with the app store
When a company is deemed an illegal monopoly, the DoJ basically becomes part of management. Antitrust settlements focus on germane elements, e.g. spin offs. But they also frequently include random terms of political convenience.
I don’t think we want a precedent where companies having a product means they have an automatic monopoly on said product.
I don't think any of what you're describing are legal "monopolies". I don't have a single Apple product in my life but I'm fairly sure there's nothing I'm prevented from doing because of that.
And back in the "Microsoft has a monopoly on IE6" ruling's days, I did not use Windows or Internet Explorer, and I was not prevented from doing anything because of that. Netscape Navigator on Linux worked fine. Sure, I occasionally hit sites that were broken and only worked in IE, but I also right now frequently hit apps that are "macOS only" (like when Claude Cowork released, or a ton of other YC company's apps).
Microsoft was found guilty, so clearly the bar is not what you're trying to claim.
Microsoft was found guilty of using their market power to do product bundling, which is illegal. The fact that they had dominance in the market is not what they got popped for, nor is it illegal.
You just described Apple.
Apple has not, to my knowledge, required OEMs to bundle Safari with macOS alongside threats to withhold macOS if they don’t comply expressly to put Firefox out of business.
But hey, maybe some weird shit happened during the clone years that I’m not privy to.
You were not prevented from doing anything, but that doesn’t mean others weren’t. For example, OEMs were not allowed to offer any other preinstalled OS as a default option. That effectively killed Be and I’m sure hindered RedHat.
Yes, but that was coupled with other factors like them strongarming vendors, already being hugely dominant on desktops and abusing that position et al. I don't see this as being the same. Maybe my bar here is wrong, but it doesn't change whether they are a monopoly or not.
That’s not how monopoly definitions work. That makes about as much sense as saying Nintendo has a monopoly on Nintendo consoles or Ford has a monopoly on Mustangs
Courts have already ruled it does in the iOS app store market. You can disagree of course but then you'd be disagreeing with legal experts who know more about anti-trust law than you do.
But Apple’s share of the desktop/laptop market is very different than their share of the mobile one.
Yes, however the parent's claim was that Apple does not have a monopoly in any market they're in which is legally demonstrably false.
Credentialism to prevent discussion of political and government entities is incredibly dangerous
You can, but that doesn't mean your opinion is as valid as those who study the subject. Otherwise we might as well believe the flat earthers.
It isn't just about monopoly or unfair competition. This can also be covered under consumer rights - the Right to Repair. No OS provider should be allowed to dictate what software you can or not run on your own device and / or OS you have paid for.
> It isn't just about monopoly or unfair competition. This can also be covered under consumer rights - the Right to Repair.
If we have a right to repair (we broadly do not, AFAICT), then that doesn't necessarily mean that we have a right to modify and/or add new functionality.
When I repair a widget that has become broken, I merely return it to its previous non-broken state. I might also decide to upgrade it in some capacity as part of this repair process, but the act of repairing doesn't imply upgrades. At all.
> No OS provider should be allowed to dictate what software you can or not run on your own device and / or OS you have paid for.
I agree completely, but here we are anyway. We've been here for quite some time.
Isn't all you have to do disable SIP?
The same way Google evaded regulatory scrutiny for refusing to allow a YouTube client for Windows Phone?
Internet Explorer Mobile is a YouTube client. You're describing a client-server disagreement when the user is talking about an entirely client-based conflict.
Google deployed custom code to actively block the clients so it went beyond just a disagreement
That's normal behavior when your server is being reverse-engineered or abused. Video bandwidth is not free.
Apple's decision is not constrained by server logic or ballooning costs, it is entirely a client-based policy to not sign CUDA drivers.
> That's normal behavior when your server is being reverse-engineered or abused. Video bandwidth is not free.
Microsoft rewrote their Windows Phone native client to pass through Google's ads. Google still blocked it.
Was it normal behavior when Google blocked Amazon Fire devices from connecting to YouTube with a web browser during the Google/Amazon corporate spat?
To be fair, Google did back down almost immediately when the tech press picked up on it.
Not allowing a native client for your monopoly market share video service on Amazon devices while also blocking Amazon's web browser on those devices is making things a bit too obvious.
I followed the instructions link and read the scripts...although the TinyGPU app is not in source form on GitHub, this looks to me like the GPU is passed into the Linux VM underneath to use the real driver and then somehow passed back out to the Mac (which might be what the TinyGrad team actually got approved).
Or I could have totally misunderstood the role of Docker in this.
https://docs.tinygrad.org/tinygpu/ are their docs, and https://github.com/tinygrad/tinygrad/tree/4d36366717aa9f1735... is the actual (user space) driver.
My read of everything is that they are using Docker for NVIDIA GPUs for the sake of "how do you compile code to target the GPU"; for AMD they're just compiling their own LLVM with the appropriate target on macOS.
As more people carry ARM laptops and keep the GPU somewhere else, I think the interesting UX question becomes whether the GPU can "follow" the local workflow instead of forcing the whole workflow to move to the GPU host. That's the problem we've been looking at with GPUGo / TensorFusion: local-first dev flow, remote GPU access when needed. Curious whether people here mostly want true attached-eGPU semantics, or just the lowest-friction way to access remote compute from a Mac without turning everything into a remote desktop / VM workflow.
I mean when it comes time to output the image from the GPU, I don't want to add a hundred milliseconds of network latency...
This is re gpu for compute not graphics.
Still undesirable latency for a lot of compute use cases, like image or video editing; it’s really only negligible for LLMs.
Since that’s definitely a big enough use case all on its own, I wonder if such a product should really just double down on LLMs.
Oh. Weird use for a graphics unit.
Using GPU for compute is nothing new or unusual these days, not for quite a while.
It’s what’s driven nearly the entire AI boom.
I'm writing scientific software that has components (molecular dynamics) that are much faster on GPU. I'm using CUDA only, as it's the eaisiest to code for. I'd assumed this meant no-go on ARM Macs. Does this news make that false?
This driver doesn't support CUDA.
This comment should be pinned at the top.
Woah, this is exciting. I'm traveling but I have a 5090 lying around at home. I'm eager to give it a go. Docs are here: https://docs.tinygrad.org/tinygpu/
I hope it'll work on an M4 Mac Mini. Does anyone know what hardware to get? You'll need a full ATX PSU to supply power, right? And then tinygrad can do LLM inference on it?
You can buy a cheap GPU enclosure for about 100$ off ali express.
Takes a standard PSU. However, Mac Minis don't have occulink. So you might be a bit limited by whatever USB C can do.
Now if Intel can get there Arc drivers in order we'll see some real budget fun.
https://www.newegg.com/intel-arc-pro-b70-32gb-graphics-card/...
32 GB of VRAM for 1000$. Plus a 500$ Mac Mini.
Those $100 ones don't come with a cage. If you do want a cage, you'll end up with $180 in total, with zero warranty.
Article mentions: "Apple finally approved our driver for both AMD and NVIDIA"
Does not mention Intel (GPUs). Select AMD GPUs work on macOS, but...
Macs (both Intel and ARM) support TB, but eGPU only work on Intel Macs, and basically only with AMD.
Good news is for medium end gaming choices are solid, and CUDA works on AMD these days.
Fortune favors the bold my friend.
I own one of these, the cage is just a piece of plastic. Anyway, I don't think 80$ is that big of a difference here. I can't really afford a 4k Nvidia GPU. Intel is my only hope.
Almost twice the price and simply more accurate info regarding price and features.
Brand is TH3P4G3. Egpu.io has decent eGPU comparisons.
I wouldn't want all that dust in my GPU fans, prefer that near my case fans. I also don't like it given I got cats and want to store/box hw. I do use the eGPU in the fuse box. If I had a larger house, I'd use a server rack.
I was recently in the market for an eGPU but for a different niche (not eGPU/eNPU/eTPU but getting a HBA via TB to connect a LTO-6 drive via SAS). I went for a Sonnet instead, very low profile and small. I also bought an Asus one. Slightly bigger, came with more fans but TB4 instead of TB3 on the Sonnet. The cages are aluminium. Those eGPU were second hand (also without warranty but quicker S&H than Chinese New Year) but came with PSU. As you also gotta buy a PSU for it which came with the eGPUs I mentioned. For me no biggie, as I got a decent PSU lying around.
Maybe I’m lacking imagination. But how will a GPU with small-ish but fast VRAM and great compute, augment a Mac with large but slow VRAM and weak compute? The interconnect isn’t powerful enough to change layers on the GPU rapidly, I guess?
> But how will a GPU with small-ish but fast VRAM and great compute, augment a Mac with large but slow VRAM and weak compute?
It would work just like a discrete GPU when doing CPU+GPU inference: you'd run a few shared layers on the discrete GPU and place the rest in unified memory. You'd want to minimize CPU/GPU transfers even more than usual, since a Thunderbolt connection only gives you equivalent throughput to PCIe 4.0 x4.
But isn’t the Mac Mini the weak link in that scenario?
It has way more unified memory than your typical dGPU.
My Mini is actually the smallest model so it actually has "small but slow VRAM" (haha!) so the reason I want the GPU for are the smaller Gemmas or Qwens. Realistically, I'll probably run on an RTX 6000 Pro but this might be fun for home.
We've seen many recent projects to stream models direct from SSD to a discrete GPU's limited VRAM on PCs.
How big a bottleneck is Thunderbolt 5 compared to an SSD? Is the 120 Gbps mode only available when linked to a monitor?
That’s what, 14GB/s? The GPU‘s VRAM can do 100x that.
A discrete consumer GPU card doesn't have enough fast RAM to run a very large model that hasn't been quanitized to hell.
That's why all the projects streaming models into the GPU from an SSD popped up recently.
“Lying around”. I’ve got an unopened 5090 in a box that I know will suffer the same fate, so I’m sending it back. So privileged to have the money to impulse buy a 5090 and yet no time to actually do anything with it.
Why does Apple need to make the drivers in a walled garden? Atleast they should support major device categories with official drivers.
Doesn't Apple support the major standard device categories: NVMe, XHCI, AHCI, and such, like most operating systems do? The challenges are all for hardware that needs a vendor-specific driver instead of conforming to a standard driver interface (which doesn't always exist). Lots of those can be supported with userspace drivers, which can be supplied by third parties instead of needing to be written by Apple.
Macs and PCs are fundamentally different. Their architectures have always been distinct though the Intel Mac era has somewhat blurred the line.
Modern Mac is Macintosh descendants and by contrast PC is IBM PC descendants (their real name is technically PC-clone but because IBM PC don’t exist anymore the clone part have been scrapped).
And with Apple silicon Mac the two is again very different, for example Mac don’t use NVMe, they use just nand (their controller part is integrated in the SoC) and they don’t use UEFI or BIOS, but a combination of Boot ROM, LLB and iBoot
> Why does Apple need to make the drivers in a walled garden?
Isn't that the whole point of the walled garden, that they approve things? How could they aim and realize a walled garden without making things like that have to pass through them?
> Why does Apple need to make the drivers in a walled garden?
For the same reason that Microsoft requires Windows driver signing?
Drivers run with root permissions.
> Why does Apple need to make the drivers in a walled garden?
Because third party drivers usually are utter dogshit. That's how Apple managed to get double the battery life time even in the Intel era over comparable Windows based offerings.
Such a shame both companies are big on vanity to make great things happen. Imagine where you could run Mac hardware with nvidia on linux. It's all there, and closed walls are what's not allowing it to happen. That's what we as customers lose when we forego control of what we purchase to those that sold us the goods.
Don't purchase? I don't own any Apple devices, everything works fine.
Unfortunately, Apple still won't release iMessage for Android or Linux (unlike every other messenger platform, like whatsapp, telegram, wechat, microsoft teams, etc, which are all cross-platform).
Because of that, you need an apple device around to be able to deal with iMessage users.
That is no longer true. https://bluebubbles.app/ Well… it’s not exactly no longer true, you do need an Apple VM but it doesn’t have to be the end device.
Why? Just make iMessage users put up with green bubbles if they want to talk to you?
Thanks to Apple co-opting phone numbers, there's literally no need to ever have iMessage for anyone
I don't understand the logic for downvotes. We vote with our wallets. When I could not update the Ram on my personal Dell machine I asked for a Frame.work in my new job. As my Intel based FW at work had thermal throttling problems, for my next personal purchase I got an AMD one. As Ubuntu had shady practices, I installed Fedora, as Gnome forced UX choices I did not want, I used KDE. As I wanted my machine to be even more stable I use an immutable spin.
The machine I'm using now represents my choices and matches what matters to me, and works closer to perfectly than all my machines in the past
And yes, I have worked with macs, and no, the UX and the entire tyranny in the Apple ecosystem was not something I could live with
And yes, this machine is fast, predictable, a joy to work with and is a tool I control, not a tool to control me. If something happens to it, I can order the part with the same price that goes into a new machine, and keep using my laptop
"We vote with our wallet, so don't complain" is a bad take in my opinion.
Like, for phones, I want a phone which runs Linux, has NFC support, and also has iMessage so my friend who only communicates with blue-bubbles and will never message a green-bubble will still talk to me. I also want it to have regulatory approval in the country I live in so I can legally use it to make calls.
Because apple has closed the iMessage ecosystem such that a linux phone can't use it, such a device is impossible. I cannot vote for it.
As such, I will complain about every phone I own for the foreseeable future.
if that's what you call a "friend"...
Interesting, but cannot run CUDA or more to the point `nvidia-smi`.
Well, to be fair, the whole shebang is from a completely different company, that have their own ML library and such, so that isn't that surprising. Although I agree that some CUDA shim or similar would be a lot more interesting, still getting to the place of running inference and training with your very own library is pretty dope already.
Pretty misleading. This driver is only for compute not graphics.
As a sizable share of the market is going to want to use this for local LLMs, I do not think this is that misleading.
GPUs can do graphics too?
Graphics was not what came to mind when I saw the headline.
Graphics is typically what comes to my mind when people talk about graphics processing units
The term eGPU gives it away, but is inaccurate.
Something like eNPU or eTPU seems more appropriate here.
If you could get Nvidia driver support on Mac’s I bet Apple would have sold more MacPro’s.
My main thought is would this allow me to speed up prompt process for large MoE models? That is the real bottleneck for m3ultra. The tokens per second is pretty good.
tinygrad does have pretty neat support for sharding things across various devices relatively easy, that'd help. I'm guessing you'd hit the bandwidth ceiling transferring stuff back and forth though instead.
What are the limitations of USB4/Thunderbolt compared with a regular PCIe slot?
I can speak to my own experience, YMMV
I hooked up a Radeon RX 9060 XT to my Feodra KDE laptop (Yoga Pro 7 14ASP9) using a Razer Core X Chroma (40Gbps), and the performance when using the eGPU was very similar to using the Radeon 880M built into the laptop's Ryzen 9 365 APU.
So at least with my setup, performance is not great at all.
On paper, TB4 is capable of pushing 5GB/s, which is somewhere between 4x and 8x of PCIe 3.0, while a 16x PCIe 4.0 link can do ~31.5GB/s.
For numbers about all PCIe generations and lane counts, see the "History and revisions" section here: https://en.wikipedia.org/wiki/PCI_Express
Edit to add: the performance I measured is in gaming workloads, not compute
Well, for starters, PCIe 5.0 x16 would do something like about 60 GB/s each way, while Thunderbolt 4 does 4 GB/s each way, TB 5 does 8 GB/s each way. If you don't actually hit the bandwidth limits, it obviously matters less. Whether you'd notice a large difference would depends heavily on the type of workload.
I think you missed a zero, TB5 does 80GB/s.
No. It does 80Gbps.
https://www.convertunits.com/from/Gbps/to/GB/s
Derp, didn’t read closely enough. Thanks
No, it does 80 Gb/s. With encoding loss it’s closer to 8GB/s
It carries pcie, but only at x4. Thunderbolt 4 is pcie gen 3 and Thunderbolt 5 is pcie gen 4.
The opportunity cost of Apple refusing to sign Nvidia's OEM AArch64 drivers is probably reaching the trillion-dollar mark, now that Nvidia and ARM have their own server hardware.
Apple got out of the server game long before they adopted aarch64, so that's a trillion worth of server hardware they never would have sold anyway. And probably not actually a trillion.
Apple was the only one stopping themselves from getting back in. It's not like the Mac is a trillion-dollar market segment to begin with.
Almost everyone including myself had MacBook Pros at my last place of work.
If Apple was in the high-end server market, I see no reason why the company I was working for would not be running macOS on Apple hardware as servers, instead of the fleet of Linux based servers they had.
USD starts sounding more and more like meaningless tokens. Billion here, trillion there. I still have 100 trillion Zimbabwean dollars somewhere.
Feels like that here in the U.S., too.