There's no way the red v2 is doing anything with a 120b parameter model. I just finished building a dual a100 ai homelab (80gb vram combined with nvlink). Similar stats otherwise. 120b only fits with very heavy quantization, enough to make the model schizophrenic in my experience. And there's no room for kv, so you'll OOM around 4k of context.
I'm running a 70b model now that's okay, but it's still fairly tight. And I've got 16gb more vram then the red v2.
I'm also confused why this is 12U. My whole rig is 4u.
The green v2 has better GPUs. But for $65k, I'd expect a much better CPU and 256gb of RAM. It's not like a threadripper 7000 is going to break the bank.
I'm glad this exists but it's... honestly pretty perplexing
It will work fine but it’s not necessarily insane performance. I can run a q4 of gpt-oss-120b on my Epyc Milan box that has similar specs and get something like 30-50 Tok/sec by splitting it across RAM and GPU.
The thing that’s less useful is the 64G VRAM/128G System RAM config, even the large MoE models only need 20B for the router, the rest of the VRAM is essentially wasted (Mixing experts between VRAM and/System RAM has basically no performance benefit).
Most people are using something in the llama family for inference. Llama server is my go to. Unsloth guides describe how to configure inference for your model of choice.
> I'm also confused why this is 12U. My whole rig is 4u.
I imagine that's because they are buying a single SKU for the shell/case. I imagine their answer to your question would be: In order to keep prices low and quality high, we don't offer any customization to the server dimensions
That's just such a massively oversized server for the number of gpus. It's not like they're doing anything special either. I can buy an appropriately sized supermicro chassis myself and throw some cards in it. They're really not adding enough value add to overspend on anything.
When the costs come down, I'll add two H100s. Until I have more work to saturate the GPUs, they're really at the limit of what I can make time to use them for. Give me a year of writing code and I'll have the need!
folks have too much money than sense, gpt-oss-120b full quant runs on my quad 3090 at 100tk/sec and that's with llama.cpp, with vllm it will probably run at 150tk/sec and that's without batching.
GPT-OSS is tailored to be extremely memory efficient. Not only is it natively using the 4.25 bit per token MXFP4 format, but it also uses sliding window attention for half of its layers. It also doesn't have that many layers, only 36 for the 120B version and 24 for the 120B version. (The 120B is also much much sparser than the 20B.)
I found a Reddit comment claiming only 36 KiB per token. With that, half a million tokens fits in 18 GB, which is less than one GPU. And three GPUs fit the parameters with room to spare (64 out of 72 GB).
Thanks for chiming in. I'm looking for a reasonably cheap local LLM machine, and multiple 3090s is exactly what I planned to buy. Do you have any recommendations or recommend any reading material before I decide to spend money on that?
edit: Found your comment about /r/localllama, but if you have anything more to add I'm still very interested.
Thanks for the confirmation, wasn't sure if I was just going a bit senile heh. Yeah, I love /r/localllama, some of the best actual practitioners of this stuff on the internet. Also, crazy awesome frankenrigs to try and get that many huge cards working together.
I was considering picking up a couple of the 48 gig 4090/3090s on an upcoming trip to China, but I just ended up getting one of the Max-Q's. But maybe the token throughput would still be higher with the 4090 route? Impressive numbers with those 3090s!
I think Hotz basically created super specific software for the gpus that throws away anything that doesn't contribute to inference (not turing complete, for example).
> There's no way the red v2 is doing anything with a 120b parameter model.
I don't see the 120B claim on the page itself. Unless the page has been edited, I think it's something the submitter added.
I agree, though. The only way you're running 120B models on that device is either extreme quantization or by offloading layers to the CPU. Neither will be a good experience.
These aren't a good value buy unless you compare them to fully supported offerings from the big players.
It's going to be hard to target a market where most people know they can put together the exact same system for thousands of dollars less and have it assembled in an afternoon. RTX 6000 96GB cards are in stock at Newegg for $9000 right now which leaves almost $30,000 for the rest of the system. Even with today's RAM prices it's not hard to do better than that CPU and 256GB of RAM when you have a $30,000 budget.
> And there's no room for kv, so you'll OOM around 4k of context.
Can't you offload KV to system RAM, or even storage? It would make it possible to run with longer contexts, even with some overhead. AIUI, local AI frameworks include support for caching some of the KV in VRAM, using a LRU policy, so the overhead would be tolerable.
Not worth it. It is a very significant performance hit.
With that said, people are trying to extend VRAM into system RAM or even NVMe storage, but as soon as you hit the PCI bus with the high bandwidth layers like KV cache, you eliminate a lot of the performance benefit that you get from having fast memory near the GPU die.
> With that said, people are trying to extend VRAM into system RAM or even NVMe storage
Only useful for prefill (given the usual discrete-GPU setup; iGPU/APU/unified memory is different and can basically be treated as VRAM-only, though a bit slower) since the PCIe bus becomes a severe bottleneck otherwise as soon as you offload more than a tiny fraction of the memory workload to system memory/NVMe. For decode, you're better off running entire layers (including expert layers) on the CPU, which local AI frameworks support out of the box. (CPU-run layers can in turn offload to storage for model parameters/KV cache as a last resort. But if you offload too much to storage (insufficient RAM cache) that then dominates the overhead and basically everything else becomes irrelevant.)"
The performance already isn't spectacular with it running all in vram. It'll obviously depend on the model: MoE will probably perform better than a dense model, and anything with reasoning is going to take _forever_ to even start beginning its actual output.
Qwen 3.5 is MoE. But you're also almost certainly running a quantized version. 120B is well over 200gb at bf16. With int4 you're looking at 60gb or so. Qwen uses relatively little kv (only about 2gb for 64k context). So you're not too snug, but if qwen isn't cutting it for you, as it didn't for me, you're kind of in a pickle. For writing tasks, int4 was simply too chaotic. I also couldn't get it to use tools.
For me, qwen didn't cut it. You're not fine tuning a 120b parameter model with 80gb. You're probably not going to be able to abliterate it either, because it's moe. Other options use more vram, and where you'd have a fair amount of buffer with qwen, you're pressed with other big models.
Honestly two rtx 8000s would probably have a better return on investment than the red v2. I have an eight gpu server, five rtx 8000, three rtx 6000 ada. For basic inference, the 8000s aren't bad at all. I'm sure the green with four rtx pro 6000s are dramatically faster, but there's a $25k markup I don't honestly understand.
What models are you testing? A 120b model with hybrid attention should fit within 80gb of VRAM fine at a 4-bit quant. Also, 4-bit quants that are done well are generally fine. They certainly don’t make the model unusable.
There's some irony in the fact that this website reads as extremely NOT AI-generated, very human in the way it's designed and the tone of its writing.
Still, this is a great idea, and one I hope takes off. I think there's a good argument that the future of AI is in locally-trained models for everyone, rather than relying on a big company's own model.
One thought: The ability to conveniently get this onto a 240v circuit would be nice. Having to find two different 120v circuits to plug this into will be a pain for many folks.
I find that the most respected writing about AI has very few signs of being written by AI. I'm guessing that's because people in the space are very sensitive to the signs and signal vs. noise.
And because people writing anything worth reading are using the process of writing to form a proper argument and develop their ideas. It’s just not possible to do that by delegating even a small chunk of the work to AI.
And usually both exist in document and lengthy communications. This gets what I wanted across with exactly my intention and then I can attach 10x length worth of AI appendix that would be helpful indexing and references.
> attach 10x length worth of AI appendix that would be helpful indexing and references.
Are references helpful when they're generated? The reader could've generated them themselves. References would be helpful if they were personal references of stuff you actually read and curated. The value then would be getting your taste. References from an AI may well be good-looking nonsense.
I agree wholeheartedly, I don’t see any balance in the effort someone dedicated to generating text vs me consuming it. If you feel there’s further insight to be gained by an llm, give me the prompt, not the output. Any communication channel reflects a balance of information content flowing and we are still adjusting to the proper etiquette.
Note, it's not one prompt (there aren't really "one prompt" any more, prompt engineering is such a 2023-2024 thing), or purely unreviewed output. It's curated output that was created by AI but iterated with me since it goes with and has to match my intention. And most of the time I don't directly prompt the agent any more. I go through a layer of agent management that inject more context into the agent that actually work on it.
Yeah, the other big benefit is that the Max-Q's have blowers that exhaust the hot air out of the box, the workstation cards would each blow their exhaust straight into the intake of the card behind it. The last card in that chain would be cooking, as the air has already been heated up by 1800W, essentially a hair dryer on high.
Or could be the server edition 6000s that just have a heatsink and rely on the case to drive air through them, those are 600W cards.
The specs show that it only has one PSU. The docs just say that it has 2 and thus needs two circuits, but I’d guess that was meant to be for the more expensive one.
Good? That's what I want out of all websites. I don't want to read what an AI believes is the best thing for a website, I want to know the honest truth.
I am a little surprised that they openly solicit code contributions with "Invest with your PRs" but don't have any statement on AI contributions.
Maybe the volume for them is ok that well-intentioned but poor quality PRs can be politely(or otherwise, culture depending) disregarded and the method of generation is not important.
That's a pretty excellent take, IMO. Just an undirected AI model doesn't do much, especially when the core team has time with the code, domain expertise, _and_ Claude.
I'm starting to think that if you have an AI repo thats basically about codegen, you should just close all issues automatically, the manually (or whatever) open the ones you/maintainers actually care about. Thats about the only way to kill some of the signal/noise ratio AIs are creating.
Then you could focus fire, like the script kiddies did with DDoS in the old days on fixing whatever preferred issues you have.
Wouldn't there be a massive duplication of effort in that case? It'll be interesting to see how the costs play out. There are security benefits to think about as well in keeping things local-first.
There are multiple efforts for 'folding at home' but for AI models at this point. I get the impression that we will see a frontier model released this year built on a system like this.
yes, this is accurate for US and “works” but it’s against code here. you’ll get mildly shocked by metallic cabinets and fixtures especially if you’re barefoot and become the new shortest path to ground.
old construction in the US sometimes did this intentionally (so old, the house didn’t have grounds. Or to “pass” an inspection and sell a place) but if a licensed electrician sees this they have to fix it.
I’m dealing with a 75 year old house that’s set up this way, the primary issue this is causing is that a 50amp circuit for their HVACs are taking a shorter path to ground inside the house instead of in the panel.
As a result the 50 amp circuit has blown through several of the common 20amp grounds and neutrals and left them with dead light fixtures and outlets because they’re bridged all over the place.
If an HVAC or two does this, I’d advise against this for your 3200 watt AI rig.
EU, you don’t want to try to energize your ground. They use step down transformers or power supplies capable of taking 115-250 (their systems are 240-250V across the load and neutral lines. Not 120 across the load and neutral like ours.)
in the US. you’re talking about energizing your ground plane with 120v and I don’t want to call that safe… but it’s REALLY NOT SAFE to make yourself the shortest path to ground on say. a wet bathroom floor. with 220V-250v.
> I’m dealing with a 75 year old house that’s set up this way
I can’t tell what practice you’re referring to. Are you perhaps referring to older wiring that connects large appliances to a neutral and two hots but no ground, e.g. NEMA 10-30R receptacles? Those indeed suck and are rather dangerous. Extra dangerous if the neutral wiring is failing or undersized anywhere.
But even NEMA 10-30R receptacles are still 120V RMS phase-to-ground. (And, bizarrely, there’s an entire generation of buildings where you might find proper 4-conductor wiring to the dryer outlet and a 10-30R installed — you can test the wiring and switch to 14-30R without any rewiring.)
The exception for residential wiring is when the neutral feed from the utility transformer fails, in which case you may have 240V phase-to-phase with the actual Earth floating somewhere in the middle (via the service’s ground connection), which can result in phase-to-neutral and phase-to-ground measured anywhere in the house varying from 0 to 240V RMS.
> wet bathroom floor
A GFCI receptacle adds a considerable degree of safety and can be installed with arbitrarily old wiring. It’s even permitted by code to install one with no ground connection as long as you label it appropriately — look it up in your local code.
it’s worse than no ground connection. There’s no neutral connection so they replaced neutrals with ground.
I believe that’s kinda naughty.
It works, but it energizes your ground plane and people do get mildly shocked. that’s making me a little nervous.
So holes have been drilled in ceilings and walls and single wire neutrals or grounds have been fished down the walls, repeatedly, by yours truly , but there’s still at least one “gfci” outlet that’s wired this way
And they’re balking at getting an electrician back out here for.
bridging neutral to ground because the neutral lines dead, uh, “works” to be technical but whoever did this moved on years ago and heaven only knows how many outlets or fixtures this was done in. I’m just finding out one by one as someone goes “hey this stopped working!”and you pull it and the neutral or ground blew like a fuse.
So that’s my whole point, this is an extremely bad idea for a 3200watt computer.
yes, they are all getting snipped and blank wall plated and marked as hazards that need to be remediated with a Dymo labeler as I discover them.
I don’t work here I just live here and have kind of a slummy owner who doesn’t want to do anything about any of it and doesn’t care if the plumbing or electrical works.
But they paid some guy like $4000 to install a totally unnecessary subpanel that’s bridging conflicting phases into the same circuits because he didn’t figure out this was what was going on. Dios Mio. I would have fixed the whole house for $1000. Miracle this hovel hasn’t burned to the ground yet.
I’m putting up with it for now but should probably bail before it does.
Late reply: I think you misunderstood my comment. I was replying to:
> It definitely comes in at a higher voltage.
The voltage supplied to a US house is 120V RMS measured phase-to-ground. You will not find a higher voltage in your house. This does not mean that it’s appropriate to run any non-negligible current from phase to the ground (green / “equipment grounding conductor”) wires.
One can get vaguely close to an accurate understanding by imagining that there are four wires coming out of your main panel: +120V, -120V, 0V white (the “actually use me” wire) and 0V green (a safety wire where any current more than a few mA or maybe tens of mA depending on application is at least a mistake). There’s no 240V to be found.
This explanation falls apart pretty quickly — the US system is AC, not DC.
If you actually use two 120V circuits that way and one breaker flips the other half will send 120V through the load back into the other circuit. So while that circuit's breaker is flipped it is still live. Very bad. Much better to use a 240V breaker that picks up two rails in the panel.
i am guessing, without any proof, that, when one breaker fails the server lose it all, or loose two GPUs, depending on whether one connected to the cpu side failed.
GPUs aren't electrically isolated from the motherboard though. An entire computer is a single unified power domain.
The only place where there's isolation is stuff like USB ports to avoid dangerous ground loop currents.
That said I believe the PSU itself provides full isolation and won't backfeed so using two on separate circuits should (maybe?) be safe. Although if one circuit tripped the other PSU would immediately be way over capacity. Hopefully that doesn't cause an extended brownout before the second one disables itself.
Yes, if you have a 240V US split-phase circuit you could make a little sub-panel with a 40A breaker feeding two 20A 120V circuits and plug the two power supplies into each side. (1600W would need a 20-A breaker because 13.3A would be too much of a 15A circuit). But it would probably make more sense to just plug them both into the same 40A 240V circuit. If you use NEMA 6-20, make sure you label it appropriately and probably color it red.
In Europe, you could plug the two power supplies into an appropriately sized 240V circuit.
In an apartment you can't rewire, you could set it up in your kitchen, which in the modern US code should have two separate 20A circuits. You will need to put it to sleep while you use appliances.
I think you're forgetting the wires? If you have one outlet with a 15-20A 120V circuit, then the wiring is almost certainly rated for 15-20A. If you just "combined" two 120V circuits into a 240V circuit, you still need an outlet that is rated for 30A, the wires leading to it also need to be rated for 30A, and it definitely needs a neutral. So you still need a new wire run if you don't have two 120V circuits right where you wanna plug in the box. To pass code you also may need to upsize conduit. If load is continuously near peak, it should be 50A instead of 30.
So basically you need a brand new circuit run if you don't have two 120V circuits next to each other. But if you're spending $65k on a single machine, an extra grand for an electrician to run conduit should be peanuts. While you're at it I would def add a whole-home GFCI, lightning/EMI arrestor, and a UPS at the outlet, so one big shock doesn't send $65k down the toilet.
Yes; I assumed 30A was minimum requirement for 240V service in US. Apparently I was wrong, 20A 240V is apparently normal. So in theory you could use a pre-existing 20A 120V circuit's wiring for a 240V (assuming it was 12/3 cable). And apparently 4-wire is now the standard for 240V service in US? Jesus we have a weird grid.
I think you might've misread GP. (Or maybe I did?)
He's not saying you would use it as two separate 120v circuits sharing a ground but rather as a single 240v circuit. His point is that it's easy to rewire for 240v since it's the same as all the other wiring in your house just with both poles exposed.
Of course you do have to run a new wire rather than repurpose what's already in the wall since you need the entire circuit to yourself. So I think it's not as trivial as he's making out.
But then at that wattage you'll also want to punch an exhaust fan in for waste heat so it's not like you won't already be making some modifications.
The wiring (at least in the US) to the 120V outlets is just one half of the split-phase 240V. If you want to send 240V down a particular wire, you can do that, by changing the breaker, but then you lose the neutral. You also make the wires dangerous to people who don't realize that the white wire is now energized at 120V over ground. (Though it's best to test to be sure anyway, as polarity gets reversed by accident, etc.) Live wires should be black or red.
I've been seeing variations on your comment a lot on HN lately and I find it a rather vapid way of looking at something so intricate as human communication. Among other things, the medium is the message!
Big companies are pushing cloud really hard, and yea the hardware prices too is a problem. People still buy Google cloud and OneDrive when they could literally pickup an old computer from trash and Frankenstein it into a NAS server.
The exabox is interesting. I wonder who the customer is; after watching the Vera Rubin launch, I cannot imagine deciding I wanted to compete with NVIDIA for hyperscale business right now. Maybe it’s aiming at a value-conscious buyer? Maybe it’s a sensible buy for a (relatively) cash-strapped ML startup; actually I just checked prices, and it looks like Vera Rubin costs half for a similar amount of GPU RAM. I’m certain that the interconnect will not be as good as NV’s.
I have no idea who would buy this. Maybe if you think Vera Rubin is three years out? But NV ships, man, they are shipping.
Sometimes you can compete with the big boys simply because they built their infra 5+ years ago and it’s not economically viable for them to upgrade yet, because it’s a multi-billion dollar process for them. They can run a deficit to run you out of the business, but if you’re taking less than 0.01% of their business, I doubt they’d give a crap.
Have to imagine each tinybox is targeting different tiers of startups trying to fine-tune/RL their way to custom models for narrow use-cases.
Maybe the target profile for exabox looks like a smaller/younger Cursor? If you're a small team with some seed funding and expertise, this kind of compute in a single box you can set up in your office feels like it could be a great fit.
The problem with all these "AI box" startups is that the product is too expensive for hobbyists, and companies that need to run workloads at scale can always build their own servers and racks and save on the markup (which is substantial). Unless someone can figure out how to get cheaper GPUs & RAM there is really no margin left to squeeze out.
Would a hedge fund that does not want to trust to a public AI cloud just buy chassis, mobos, GPUs, etc, and build an equivalent themselves? I suspect they value their time differently.
We may be surprised how illefficient companies are in organizing the creation of sophisticated things (including processes) for themselves, to use (so for the cost center column).
Higher management figures out things to do in strategic level, in brief, and pushes on "soldiers", who kick it through in the least time (cheapest of the cheapest, for the sake of the quarterlies) EXACTLY the way management told it. Because they have to, their job is to make happen the company objectives given, the way it is given. Pushing out crap in the shape of the thing expected.
Larger organiztaion can use these kind of things the most. Even if they don't do that.
Ollama or llama.cpp are also common alternatives. But a 8B model isn't going to have much real-world knowledge or be highly reliable for agentic workloads, so it makes sense that people will want more than that.
the compute density is insane. but giving a 70B model actual write access locally for agentic workloads is a massive liability. they still hallucinate too much. raw compute without strict state control is basically just a blast radius waiting to happen.
$12,000 for the base model is insane. I have an Apple M3 Max with 128GB RAM that can run 120B parameter models using like 80 watts of electricity at about 15-20 tokens/sec. It's not amazing for 120B parameter models but it's also not 12 grand.
It is very comparable if you work out the $/tok/s on inference. I did some napkin math and it looks like you’re getting roughly 3x the performance for 3x the cost. Red v2 vs Mac Studio M3 Ultra 96GB.
If you compare tokens/kWh efficiency then my math has Mac Studio being about 1.5x more efficient.
M3 has tolerable decode performance for the price, and that's what people would care about most of the time. they underperform severely wrt. prefill, but that's a fraction of the workload. AI, even agentic AI, spends most of its time outputing tokens, not processing context in bulk.
it's for fools. i bought 160gb of vram for $1000 last year. 96gb of p40 VRAM can be had for under $1000. And it will run gpt-oss-120b Q8 at probably 30tk/sec
P40 is Tesla architecture which is no longer receiving driver or CUDA updates. And only available as used hardware. Fine for hobbyists, startups, and home labs, but there is likely a growing market of businesses too large to depend on used gear from ebay, but too small for a full rack solution from Nvidia. Seems like that's who they're targeting.
I suppose if I rent a cloud GPU and just let it sit there dark and do nothing then I wouldn't have to move any data to it. Otherwise, I'm uploading some kind of work for it to do. And that usually involves some data to operate on. Even if it's just prompts.
So you also believe when you rent a server you are sharing your data with the cloud? AWS and GCP are copying all private data on servers? Give me a break. There's a big difference between renting a server and using an API.
> So you also believe when you rent a server you are sharing your data with the cloud [hosting provider]?
Only if you upload your data to that cloud server you rented. Then, by definition, you are.
> AWS and GCP are copying all private data on servers?
Every computer copies data when moving it. Several times, in fact. Through network card buffers, switches, system memory, disk caches, and finally to some form of semi-permanent storage.
I don't have to think Amazon is stealing my data to be aware that Amazon S3 buckets containing privileged information are routinely found open. I don't have to think that Google is spying on me to know that operating equipment my business owns on prem and does not share requires me to trust fewer people and less complex systems than doing the same work from the cloud.
You are very quick to make foolish assumptions and assign them to others.
> In order to keep prices low and quality high, we don't offer any customization to the box or ordering process. If you aren't capable of ordering through the website, I'm sorry but we won't be able to help.
Has this guy never worked on a B2B product before? Nobody is going to order a $10 million piece of infrastructure through your website's order form. And they are definitely going to want to negotiate something, even if it's just a warranty. And you'll do it because they're waving a $10 million check in your face.
The tone of this website is arrogant to the point of being almost hostile. The guy behind this seems to think that his name carries enough weight to dictate terms like this, among other things like requiring candidates to have already contributed to his product to even be considered for a job. I would be extremely surprised if anyone except him thinks he's that important.
I haven’t seen tinygrad used for any mainstream production project or thing of value, yet.
Besides a lot of self congratulatory pats on the back for how elegant it is. Honestly, when I read it, it looked confusing as all the other ML libraries. Not actually simple like Karpathy’s stuff.
All that to say, I do really want it to succeed. They should probably hire some practical engineers and not just guys and gals congratulating themselves how elegant and awesome they are.
Filling out an onboarding form is an example of what he's not willing to do, not the only thing he isn't willing to do.
> we don't offer any customization to the box or ordering process
Every B2B deal of that size that I've ever seen requires at least weeks of meetings between the customer and vendor, in which every detail is at least discussed if not negotiated. That would certainly constitute a "customization" to this guy's prescribed ordering process, which is to "Buy it now" [1] through the website at the stated price like you're ordering a jar of peanuts on Amazon. This is not "framing", it's what the guy said. If it isn't what he meant then he needs to fix his copy.
[1] Yes, there is an actual "Buy it now" button for a $65,000 business purchase that takes you to a page that looks just like a Stripe form. There isn't even a textbox for delivery instructions. Wild.
> Then if they succeed, I guess you're going to see a different process for the first time in your life.
Sure, I guess. Far more likely that they won't succeed, and it will be because of their pointless refusal to cooperate with others. I'm curious why you think we should "disrupt" companies putting a little due diligence into massive purchases.
> On a website where we frequently talk about disruptive business models, this whole attitude kinda stinks.
I could say the same thing about making a comment like this on a website where groupthink is rightfully mocked.
> you're going to see a different process for the first time in your life
That sounds very neutral, but wouldn't this, by removing the human element and flexibility from business transactions, be a further step along a general enshittification trend?
There isn't a $10MM device right now, just $64M and under. I doubt the order process will remain the same in 12 months when the $10MM device becomes available
It's a shipping container. Look at the dimensions. They say concrete slab probably half as a joke, half because building code would require it to consider it a non-temporary structure.
Are you referring to the images of branded shipping containers on their Twitter page that have visible Gemini watermarks … and jokes in the comments about AI trailer parks?
20x8x8.5 ft is the dimensions of a half shipping container. You think that render is a joke but it's not. They don't have photos yet because it's a 2027 product (if it actually comes out which I would bet against).
It's also funny that they explicitly list driver quality as "good" for the base option and "great" for the intermediate one. You're really going to deliberately provide worse drivers for the machine I paid you for, just because I didn't buy the more expensive one?
I mean I'm sure lots of companies do this in practice because tickets for higher-paying customers naturally get prioritized, but directly stating your intention to do it on your home page is hilarious.
Nvidia drivers are better than AMD. It's not really something they have control over. Geohot is definitely obsessed with bitching about driver bugs though.
It doesn't matter if it's a joke. The non-technical manager or VP making this purchase will not understand it and will expect poor treatment from this vendor, an expectation that will be reinforced by numerous other things on this page. There is no reason to include it at all.
To me it signals honesty. But this is a subjective judgement. It really sounds like you subjectively disliked the page, and you're trying to present that dislike as objective fact. It really annoyed me the way you kept changing your argument to justify that. Why not just say "I dislike their marketing copy, it rubbed me the wrong way" and leave it at that?
I'm confused by this comment. Is it not obvious that everything I've said is my opinion?
Not everyone feels the need to hedge everything they say with "to me..." and "it really sounds like...".
> It really annoyed me ...
I have no idea what you expect me to do with that information.
> you kept changing your argument
I'm not sure what you're referring to. Are you talking about the joke/not-a-joke thing? I didn't change my argument, I dismissed a shallow objection to an irrelevant detail. My point is that regardless of why the driver quality was included on the page, customers are going to take it the wrong way. Yes, that is my opinion, because apparently that needs to be explicitly stated.
Everyone else who disagreed with me seems to have understood all this so I don't know what the source of your confusion is.
> It seems that you work a lot with managers who have no clue what they are buying and why.
There are certain quirks of this platform's user base that always make me laugh. For example, HNers absolutely love to imply something condescending about the other guy's workplace in order to make their point.
Watch this, I can do it too: Working with managers who make $65,000 (or $10 million) purchases with no more due diligence than reading a marketing page and clicking "Buy it now" is not the flex you think it is.
I was involved in it-related deals on both purchasing and selling sides. Sums involved were larger than both numbers you mentioned.
And I honestly see almost no correlation between the amount of negotiation involved, and value received.
Some of the most useful things we've integrated were either free or meant that only the "buy it now" button had to be clicked.
Some of the absolutely worst systems I had to work with were purchased after making a call to that "let us know" number.
This tiny guy is mostly saying that he doesnt have the time for enterprise bla-bla. I am not sure he can organise enterprise sales with this attitude but can definitely relate to it!
Yeah it’s a little odd. Maybe they are meant to be really really cool toys? People regularly spend more than $65k on things like cars to show off, so it could be like that.
I have no use for these but I might buy one anyway if I won the lottery. ¯\_(ツ)_/¯
Tinybox is cool but I think the market is maybe looking more for a turn-key explicit promise of some level of intelligence @ a certain Tok/s like "Kimi 2.5 at 50Tok/s".
inference speed is like monitor Hz; sure, you go from 60 to 120Hz and thats noticeable, but unless your model is AGI, at some point you're just generating more code than you'll ever realistically be able to control, audit and rely on.
So, context is probably more $/programming worth than inference speed.
Perhaps this company should think about acting as a landlord for their hardware. You buy (or lease) but they also offer colocation hosting. They could partner with crypto miners who are transitioning to AI factories to find the space and power to do this. I wonder if the machines require added cooling, though, in what would otherwise be a crypto mining center. CoreWeave made the transition and also do colocation. The switchover is real.
I think Tinygrad should think about recycling. Are they planning ahead in this regard? Is anyone?
My thought is if there was a central database of who own what and where, at least when the recycling tech become available, people will know where to source their specific trash (and even pay for it.) Having a database like that in the first place could even fuel the industry.
I was more worried by the 600kW power requirement... that's 200 houses at full load (3kw) in southern europe... which likely means 400 houses at half load.
the town near my hometown has 650 – 800 houses (according to chatgpt).
I think the above commentor is reflecting on the total energy use from having a 600KW load running 24/7. I suppose the more interesting observation is the 14 MWh of daily consumption, enough to charge 100 Rivians every day.
> and heat pump cooling / heating became the norm.
We're not all solidly middle-class (especially in Southern and Eastern Europe) and as such we cannot afford those heat pumps. But we'll have to eat the increased energy costs brought by insane server configurations like the ones from the article, so, yeey!!!
I would love to see real-life tokens/sec values advertised for one or various specific open source models.
I'm currently shopping for offline hardware and it is very hard to estimate the performance I will get before dropping $12K, and would love to have a baseline that I can at least always get e.g. 40 tok/s running GPT-OSS-120B using Ollama on Ubuntu out of the box.
Look for llmfit on github. This will help with that analysis. I've found it reasonably accurate. If you have Ollama already installed, it can download the relevant models directly.
Not sure why they stopped using 6 GPUs in thei builds - with 4 GPUs, both 9070 and rtx6000 come in 2 slot designs, so it easy to build it yourself using a bit more expensive, but still fairly regular motherboard.
With 6 GPUs you have to deal with risers, pcie retimers, dual PSUs and custom case for so value proposition there was much better IMO
Cool that you have a dual power supply model. It says rack mountable or free standing. Does that mean two form factors? $65K is more than we can afford right now but we are definitely eventually in the market for something we can run in our own colo.
It's funny though... we're using deepseek now for features in our service and based on our customer-type we thought that they would be completely against sending their data to a third-party. We thought we'd have to do everything locally. But they seem ok with deepseek which is practically free. And the few customers that still worry about privacy may not justify such a high price point.
Most privacy talk folds on contact with a quote. Latency and convenience beat philosophy fast once someone wants a dashboard next week, and a lot of "data sensitivity" talk is just the corporate version of buying "organic" food until the price tag shows up.
If private inference is actually non-negotiable, then sure, put GPUs in your colo and enjoy the infra pain, vendor weirdness, and the meeting where finance learns what those power numbers meant.
The real case for private inference is not "organic", it's "slow food". Offering slow-but-cheap inference is an afterthought for the big model providers, e.g. OpenRouter doesn't support it, not even as a way of redirecting to existing "batched inference" offerings. This is a natural opening for local AI.
Regarding 2x faster than pytorch being a condition for tinygrad to come out of alpha:
Can they/someone else give more details as to what workloads pytorch is more than 2x slower than the hardware provides? Most of the papers use standard components and I assume pytorch is already pretty performant at implementing them at 50+% of extractable performance from typical GPUs.
If they mean more esoteric stuff that requires writing custom kernels to get good performance out of the chips, then that's a different issue.
$12,000 gets you 1Gb/s networking and vanilla Ubuntu 24.04. Napkin math on the hardware it looks like margins are around 50% which feels like a school fundraiser where everyone pays what is obviously way more than normal retail price for X because "it's for the children."
I'm not sure what tinygrad is but I assume the markup is because the customer is making a conscious choice to support the tinygrad project. But what's unusual is there is apparently no reason whatsoever to buy this hardware, even if you plan on using tinygrad exclusively for your project. At least with System76 hardware I get (in theory) first class support for Pop!_OS.
Curious to know who will spend this much money without external funding? Would you spend any VC invested money into this nameless brand? Are there any guardrails or clauses to protect the kind of expenses?
Depends. If token speed isn't a big deal, then I think strix halo boxes are the meta right now, or Mac studios.
If you need speed, I think most people wind up with something like a gaming PC with a couple 3090 or 4090s in it.
Depending on the kinds of models you run (sparse moe or other), one or the other may work better.
Performance (tok/s and PP) or quality (model size)? Pick one.
In terms of GPU memory bandwidth (models fitting in the ~48GB of RTX 5000 Pro card), the RTX card I described above has over 2x the bandwidth of an M5 Max.
If leveraging system RAM (the 128GB-256GB outside the GPU) to run larger models, then the memory bandwidth is ~6x slower than M5 Max.
For models fitting in the ~48GB RTX memory, like dense Qwen3.5 27B models, the RTX will be 2-4x faster than M5 Max. For models that don't fit in the 48GB RTX memory, the M5 Max will be 5-20x faster.
Also worth considering future upgrades: Do you plan to throw away the machine in a few years, or pick up multiple used RTX 6000 Pro cards when people start ditching them?
Sadly $5k is sort of a no-man's land between "can run decent small models" and "can run SOTA local models" ($10k and above). It's basically the difference between the 128GB and 512GB Mac Studio (at least, back when it was still available).
The DGX Spark is probably the best bang for your buck at $4k. It's slower than my 4090 but 128gb of GPU-usable memory is hard to find anywhere else at that price. It being an ARM processor does make it harder to install random AI projects off of GitHub because many niche Python packages don't provide ARM builds (Claude Code usually can figure out how to get things running). But all the popular local AI tools work fine out of the box and PyTorch works great.
DGX Spark is a fantastic option at this price point. You get 128GB VRAM which is extremely difficult to get at this price point. Also it’s a fairly fast GPU. And stupidly fast networking - 200gbps or 400gbps mellanox if you find coin for another one.
I’m not very well versed in this domain, but I think it’s not going to be “VRAM” (GDDR) memory, but rather “unified memory”, which is essentially RAM (some flavour of DDR5 I assume). These two types of memory has vastly different bandwidth.
I’m pretty curious to see any benchmarks on inference on VRAM vs UM.
I’m using VRAM as shorthand for “memory which the AI chip can use” which I think is fairly common shorthand these days. For the spark is it unified, and has lower bandwidth than most any modern GPU. (About 300 GB/s which is comparable to an RTX 3060.)
So for an LLM inference is relatively slow because of that bandwidth, but you can load much bigger smarter models than you could on any consumer GPU.
Meh. DGX is Arm and CUDA. Strix is X86 and ROCm. Cuda has better support than ROCm . And x86 has better support than Arm.
Nowadays I find most things work fine on Arm. Sometimes something needs to be built from source which is genuinely annoying. But moving from CUDA to ROCm is often more like a rewrite than a recompile.
> But moving from CUDA to ROCm is often more like a rewrite than a recompile.
Isn't everyone* in this segment just using PyTorch for training, or wrappers like Ollama/vllm/llama.cpp for inference? None have a strict dependency on Cuda. PyTorch's AMD backend is solid (for supported platforms, and Strix Halo is supported).
* enthusiasts whose budget is in the $5k range. If you're vendor-locked to CUDA, Mac Mini and Strix Halo are immediately ruled out.
Most everything starts as PyTorch. (Or maybe Jax.) But the inference engines all use hand tuned CUDA kernels - at least the good ones do. You have to do that to optimize things.
I'm certain inference engines don't use hand-tuned CUDA on Radeon or Mac Mini chips. My statement holds: those engines have no strict dependency on CUDA, or they'd be Nvidia-only.
Biggest Mac Studio you can get. The DGX Spark may be better for some workflows but since you're interested in price, the Mac will maintain it's value far longer than the Spark so you'll get more of your money out of it.
With $5k you have to make compromises. Which compromises you are willing to make depends on what you want to do - and so there will be different optimal setup.
Machines with the 4xx chips are coming next month so maybe wait a week or two.
It's soldered LPDDR5X with amd strix halo ... sglang and llama.cpp can do that pretty well these days. And it's, you know, half the price and you're not locked into the Nvidia ecosystem
The most interesting part of Tinybox isn't just the hardware, but the push for a more vertical integration with tinygrad. We've become so accustomed to the CUDA/PyTorch stack that seeing a serious attempt at a different software-hardware synergy is refreshing, even if the hardware specs or price point relative to DIY homelabs raise some eyebrows for power users. It's more about reducing the friction for researchers who want a "just works" environment without the nightmare of driver/toolkit version hell.
Would be very curious how RL benchmarks shake out vs M5 Pro/Max.
Doubt local inference is the target use case near nearly as much as post-training. I could totally see something like this being super appealing for a startup looking to do some fine-tuning/distillation to tune a small open-weight model for a narrow use case.
It might be a bit CPU and RAM starved… Which in theory should be OK, but in practice you'll find production workloads that struggle because of this. Just make sure whatever you want to run on this is indeed extremely GPU-bound, or you might have bad surprises later.
I thought the most interesting thing about tinygrad was that theoretically you could render a model all the way into hardware similar to Taalas (tinygrad might be where Taalas got the idea for all I know).
I could swear I filed a GitHub issue asking about the plans for that but I don't see it. Anyway I think he mentioned it when explaining tinygrad at one point and I have wondered why that hasn't got more attention.
As far as boxes, I wish that there were more MI355X available for normal hourly rental. Or any.
I just don’t believe that this can run inference on a 120 billion parameter model at actually useful speeds.
Obviously any Turing machine can run any size of model, so the “120B” claim doesn’t mean much - what actually matters is speed and I just don’t believe this can be speedy enough on models that my $5000 5090-based pc is too slow for and lacks enough vram for.
The "to be fair" part of my comment was saying that the tinygrad website doesn't claim 120B.
Also nobody is comparing this box to an $10M nVidia rack scale deployment. They're comparing it to putting all of the same parts into their Newegg basket and putting it together themself.
Tinygrad devices are interesting, I wish I have screen captures - but their prices have gone up and some specs like RAM have gone down.
A single box with those specs without having to build/configure (the red and green) - I could see being useful if you had $ and not time to build/configure/etc yourself.
I have 8x RTX 6000 Pro. Better to run the 300 W version of the cards. And it costs close to their 4x version. I get why they make it so big. So you can cool it at home. I prefer to just put in datacenter. Much cheaper power.
Why do I get the impression that I get more bang for the buck by going through OpenRouter? Of course, not anyone can do that and there are security and other concerns.
Looking at these prices I’m just thinking that as a user it makes no sense to buy this when you can just use the subsidised stuff from AI companies and then buy it a few years later at a tiny % of the cost.
its a bit weird to me ud need to be contributor to their software to work in operations or hardware, but I suppose its ok for tinycompany. in long term its likely better to have domain experts and not bias everything towards the same thing.
the boxes look cool but how good are they really? the cheapest box seems pricey at 12 for a what is essentially a few gaming gpus. i dont see why you couldnt make that like half the price. u could do a PC/server build thats much much faster for way less. size doesnt matter if its more than twice the price i think...
the more expensive box has atleast real processing gpus but afaik also not very popular ones, this one seems maybe more fair priced (there seems a big difference in bang for buck between these???).
the third one suggested looks like a joke.
dont get me wrong, this seems like a really cool idea. But i dont see it taking off as the prices are corporate but the product seems more home use.
maybe in time they will find a better balance, i do respect the fact that the component market now is sour as hell and making good products with stable prices is pretty much i possible.
id love one of these machines someday, maybe when i am less poor, or when they are xD.
(love the styling of everything, this is the most critical i could be from a dumb consumer perspective, which i totally am btw.)
These things don’t have Flash Attention or either have a really hacked together version of it. Is it viable for a hobby? Sure. Is it viable for a serious workload with all the optimizations, CUDA, etc.. Not really.
I wonder if this is frontpage right now because of the other tiiny (the names are similar) video that went viral ... which turns out wasn't an actual product by the tinygrad linked in this post[1]
They answered your question with a pretty specific uptime target. Calling it a dodge and then moving the goalposts with a new question as your follow up doesn’t speak to you acting in good faith.
Theres a lot there that makes sense & I think needs to be considered. But a lot just seems to be out of the blue, included without connection, in my view. Feels like maybe are in-grouo messages, that I don't understand. How this is headered as against democracy is unclear to me, and revolting. I both think we must grapple with the world as it is, and this post is in that area, strongly, but to let fear be the dominant ruling emotion is one of the main definitions of conservativism, and it's use here to scare us sounds bad.
Geohotz's politics are fairly straightforward once you understand his background. Geohotz is the prodigy child who, at the age of ~16 accomplished amazing technical feats on his own.
And his politics are a derivative of Great Man Theory, and his positions on things like democracy follow from that. This idea, and those espoused by some of the VC/tech elite like Peter Theil are that singular hardworking genius individuals can change the world on their own, and everyone who not in this top 0.1% are borderline NPCs.
They do this both because of their genius/hardwork, and also because they are willing to break the rules that are set forth by this bottom 99.9%.
I'm starting to call this ideology Authoritarian techno-Libertarianism. Its a delibriately oxymoronic name that I use, because these "Great Men" are definitely trying to change the world. IE, they are trying to impose their goals and values on the world without getting the buyin of other people.
Thats the "authoritarian" part. And then the "libertarian" part is that they are going about this imposition of their will on the world by doing it all themselves, through their own hard work.
Think "Person invents a world changing technology, that some people thing is bad, and just releases it open source for anyone to use". AI models are a great example, in fact. Once that technology is out there the genie cannot be put back into the bottle and a ton of people are going to lose their jobs, ect.
A distain for democracy follows directly from things like this. You dont wait for people to vote to allow you to change the world by inventing something new. You just do and watch the results.
> also because they are willing to break the rules that are set forth by this bottom 99.9%[...] they are going about this imposition of their will on the world by doing it all themselves, through their own hard work.
I think all these wildly successful neo-feudalists get increasingly emboldened the more they get away with bigger and bigger social infractions.
It's also clear that they haven't experienced existed an environment with extreme inequality - it's not safe for anyone there! They think the NPC plebs will continue to follow "the rules" ad perpetuam without considering that it is a direct result of the stability they are actively undermining. They clearly don't read enough history.
What makes it “Libertarianism” still? To me it feels like they’re taking away freedom, control, and influence from everyone who is not them. Even the concentration of wealth is itself taking away everyone else’s places in the world.
Its libertarian because it is fundamentally about individuals acting on their own without going through the government, ect. It is an individualistic framework. Individuals going about achieving their goals, even through powerful corporations, falls squarely within what libertarians support.
Yes, you can make some philosophical point about how if corporations are powerful enough, how is that in any way different from governments.
But, powerful corporations controlling society, in some sort of fallout style or bioshock style dystopia clearly describes a libertarian dystopia, not a left leaning or even fascist dystopia.
He had a video on Youtube where he proudly gloated about how he voted for Trump in not one but two elections, how happy he is that he can now openly talk about it, how its a fresh start for US, how catastrophic Harris would have been.
Did he take down the video because of embarrassment or did he fear negative impact on his sales?
For those unaware, Mencius Moldbug is the pen name of Curtis Yarvin, thought leader for the Silicon Valley branch of right-wing technofascist weirdos which includes Peter Thiel and apparently half of a16z.
That thing is NOT related to tinybox or tinygrad in any way. It is basically copyright infringement. Unless you’re astroturfing here I suggest you get your money back.
Sorry for even mentioning astroturfing, haha. It’s just because the promotion of the device is based on trying to fool people it was made by tiny corp.
Yeah, installing random wheels from non official sources is an improvement. Not sure I’d characterize that as an unmitigated win. But also as soon as you try to do more involved things, at least personally, I ran into serious challenges getting things to work.
Finally, a computer that should be able to run Monster Hunter Wilds with decent performance.
But let’s be real, 12k is kinda pushing it - what kind of people are gonna spend $65k or even $10M (lmao WTAF) on a boutique thing like this. I dont think these kinds of things go in datacenters (happy to be corrected) and they are way too expensive (and probably way too HOT) to just go in a home or even an office “closet”.
But how will I make ad-supported youtube videos about how I automated my life with OpenClaw using a $10M boutique AI server to make a few thousand in ad revenue while burning tens of thousands per month on API cost.
These specs look enormously cheaper than doing it with dell servers. The last quote I had for a bog standard dell server was $50k and only if bought in the next few days or so. The prices are going up weekly.
These are "unsupported" configurations. Nvidia/AMD discourage running multiple gaming/workstation cards and encourage customers to buy $500K SXM/OAM servers.
Since when did our perception of tiny blow out of size in tech? Is it the influence of "hello world" eletron apps consuming 100mb of mem while idle setting the new standard? Anyway being an AI bro seems like an expensive hobby...
> Can I pay with something besides wire transfer?
In order to keep prices low and quality high, we don't offer any customization to the box or ordering process. Wire transfer is the only accepted form of payment.
Yeah I agree the FTC article could be more clear here. I think they call out Western Union because those are tools that are commonly used by scammers.
But let’s be clear: the risks are the same if you are wiring money through Western Union or wiring through any other bank. Once you wire the money you do not have the same protections as other payment mechanisms. And if you don’t get the product as described, you are likely out your money. This is compared to other forms of payment like credit cards where you are protected. With a credit card you can issue a charge back to the seller and get your money back in the case of fraud. With a wire transfer you cannot.
Tbh my experience is in the non AI uses, recently I was looking at Gaussian splatting tools and it seemed the majority of it was CUDA only. I’m also still bothered AMD for ages claimed my card (5700xt) would be getting rocm but just abandoned it.
But even in the amd stack things (like ck and aiter) consumer cards are not even second class citizens. They are a distance third at best.
If you just want to run vllm with the latest model, if you can get it running at all there are going to be paper cuts all along the way and even then the performance won't be close to what you could be getting out of the hardware.
They MIGHT pay you IF you're a fit. They're bounties, i.e. spec work. They also pay a max of $1000, most of them significantly less. You can see more info at the link in that line:
> All bounties paid out at my (geohot) discretion. Code must be clean and maintainable without serious hacks.
No thanks. If you want to try before you buy, have your candidates do a paid test project. Founders need to stop acting like it's a privilege to work for them. Any talent worth hiring has plenty of other options that will treat them with respect.
There's no way the red v2 is doing anything with a 120b parameter model. I just finished building a dual a100 ai homelab (80gb vram combined with nvlink). Similar stats otherwise. 120b only fits with very heavy quantization, enough to make the model schizophrenic in my experience. And there's no room for kv, so you'll OOM around 4k of context.
I'm running a 70b model now that's okay, but it's still fairly tight. And I've got 16gb more vram then the red v2.
I'm also confused why this is 12U. My whole rig is 4u.
The green v2 has better GPUs. But for $65k, I'd expect a much better CPU and 256gb of RAM. It's not like a threadripper 7000 is going to break the bank.
I'm glad this exists but it's... honestly pretty perplexing
It will work fine but it’s not necessarily insane performance. I can run a q4 of gpt-oss-120b on my Epyc Milan box that has similar specs and get something like 30-50 Tok/sec by splitting it across RAM and GPU.
The thing that’s less useful is the 64G VRAM/128G System RAM config, even the large MoE models only need 20B for the router, the rest of the VRAM is essentially wasted (Mixing experts between VRAM and/System RAM has basically no performance benefit).
Could you share what you are using for inference and how you are running it? I have a 64G VRAM/128G system RAM setup.
Most people are using something in the llama family for inference. Llama server is my go to. Unsloth guides describe how to configure inference for your model of choice.
Split RAM and GPU impacts it more than you think. I would be surprised if the red box doesn’t outperform you by 2-3X for both PP and TG
Yeah I've got the q4 gpt-oss-120b running at ~40-60 tokens per second on an M5 Pro.
> I'm also confused why this is 12U. My whole rig is 4u.
I imagine that's because they are buying a single SKU for the shell/case. I imagine their answer to your question would be: In order to keep prices low and quality high, we don't offer any customization to the server dimensions
That's just such a massively oversized server for the number of gpus. It's not like they're doing anything special either. I can buy an appropriately sized supermicro chassis myself and throw some cards in it. They're really not adding enough value add to overspend on anything.
The major selling point of the tinyboxes is that you're able to run them in your office without any hassle.
I used to own a Dell Poweredge for my home-office, but those fans even on minimal setting kept me up at night
Was that cheaper than a Blackwell 6000?
But yeah, 4x Blackwell 6000s are ~32-36k, not sure where the other $30k is going.
I bought the A100s used for a little over $6k each.
Oh, why'd you go that route? Considering going beyond 80 gigs with nvlink or something?
When the costs come down, I'll add two H100s. Until I have more work to saturate the GPUs, they're really at the limit of what I can make time to use them for. Give me a year of writing code and I'll have the need!
folks have too much money than sense, gpt-oss-120b full quant runs on my quad 3090 at 100tk/sec and that's with llama.cpp, with vllm it will probably run at 150tk/sec and that's without batching.
> gpt-oss-120b full quant runs on my quad 3090
A 120B model cannot fit on 4 x 24GB GPUs at full quantization.
Either you're confusing this with the 20B model, or you have 48GB modded 3090s.
Some of you folks on here love to argue, gpt-oss-120b was trained in 4 bits, so it pretty much takes up 60gb.
Good point, but you still need KV cache and more. Fitting the model alone to RAM doesn’t get the job done.
GPT-OSS is tailored to be extremely memory efficient. Not only is it natively using the 4.25 bit per token MXFP4 format, but it also uses sliding window attention for half of its layers. It also doesn't have that many layers, only 36 for the 120B version and 24 for the 120B version. (The 120B is also much much sparser than the 20B.)
I found a Reddit comment claiming only 36 KiB per token. With that, half a million tokens fits in 18 GB, which is less than one GPU. And three GPUs fit the parameters with room to spare (64 out of 72 GB).
Yeah, it doesn't take much. I'm looking at it right now, KV cache is about 4gb of vram, compute buffer =~ 1.5gb at full 128k context.
Thanks for chiming in. I'm looking for a reasonably cheap local LLM machine, and multiple 3090s is exactly what I planned to buy. Do you have any recommendations or recommend any reading material before I decide to spend money on that?
edit: Found your comment about /r/localllama, but if you have anything more to add I'm still very interested.
You're almost certainly (definitely, in fact) confusing the 120b and 20b models.
I'm most certainly not doing so.
Oh I missed the "quad" before 3090.
How're you fitting a model made for 80 gig cards onto a GPU with 24 gigs at full quant?
MoE layers offload to CPU inference is the easiest way, though a bit of a drag on performance
Yeah, I'd just be pretty surprised if they were getting 100 tokens/sec that way.
EDIT: Either they edited that to say "quad 3090s", or I just missed it the first time.
you are correct, I did forget to add quad. you should join us in r/localllama
check out what other people are getting. you're welcome.
https://www.reddit.com/r/LocalLLaMA/comments/1nunq7s/gptoss1... https://www.reddit.com/r/LocalLLaMA/comments/1p4evyr/most_ec...
Thanks for the confirmation, wasn't sure if I was just going a bit senile heh. Yeah, I love /r/localllama, some of the best actual practitioners of this stuff on the internet. Also, crazy awesome frankenrigs to try and get that many huge cards working together.
I was considering picking up a couple of the 48 gig 4090/3090s on an upcoming trip to China, but I just ended up getting one of the Max-Q's. But maybe the token throughput would still be higher with the 4090 route? Impressive numbers with those 3090s!
What's the rig look like that's hosting all that?
He said quad 3090 not single
Yeah, pretty sure that was edited in after I commented because 150 toks/sec was also new, but could’ve just missed it.
I think Hotz basically created super specific software for the gpus that throws away anything that doesn't contribute to inference (not turing complete, for example).
> There's no way the red v2 is doing anything with a 120b parameter model.
I don't see the 120B claim on the page itself. Unless the page has been edited, I think it's something the submitter added.
I agree, though. The only way you're running 120B models on that device is either extreme quantization or by offloading layers to the CPU. Neither will be a good experience.
These aren't a good value buy unless you compare them to fully supported offerings from the big players.
It's going to be hard to target a market where most people know they can put together the exact same system for thousands of dollars less and have it assembled in an afternoon. RTX 6000 96GB cards are in stock at Newegg for $9000 right now which leaves almost $30,000 for the rest of the system. Even with today's RAM prices it's not hard to do better than that CPU and 256GB of RAM when you have a $30,000 budget.
> And there's no room for kv, so you'll OOM around 4k of context.
Can't you offload KV to system RAM, or even storage? It would make it possible to run with longer contexts, even with some overhead. AIUI, local AI frameworks include support for caching some of the KV in VRAM, using a LRU policy, so the overhead would be tolerable.
Not worth it. It is a very significant performance hit.
With that said, people are trying to extend VRAM into system RAM or even NVMe storage, but as soon as you hit the PCI bus with the high bandwidth layers like KV cache, you eliminate a lot of the performance benefit that you get from having fast memory near the GPU die.
> With that said, people are trying to extend VRAM into system RAM or even NVMe storage
Only useful for prefill (given the usual discrete-GPU setup; iGPU/APU/unified memory is different and can basically be treated as VRAM-only, though a bit slower) since the PCIe bus becomes a severe bottleneck otherwise as soon as you offload more than a tiny fraction of the memory workload to system memory/NVMe. For decode, you're better off running entire layers (including expert layers) on the CPU, which local AI frameworks support out of the box. (CPU-run layers can in turn offload to storage for model parameters/KV cache as a last resort. But if you offload too much to storage (insufficient RAM cache) that then dominates the overhead and basically everything else becomes irrelevant.)"
The performance already isn't spectacular with it running all in vram. It'll obviously depend on the model: MoE will probably perform better than a dense model, and anything with reasoning is going to take _forever_ to even start beginning its actual output.
I know llama.cpp can, it certainly improved performance on my RAM-starved GPU.
This does not match my experience with 120B~ models. I run Qwen3.5 122b A10B on about 80GB of vRAM just fine.
Qwen 3.5 is MoE. But you're also almost certainly running a quantized version. 120B is well over 200gb at bf16. With int4 you're looking at 60gb or so. Qwen uses relatively little kv (only about 2gb for 64k context). So you're not too snug, but if qwen isn't cutting it for you, as it didn't for me, you're kind of in a pickle. For writing tasks, int4 was simply too chaotic. I also couldn't get it to use tools.
For me, qwen didn't cut it. You're not fine tuning a 120b parameter model with 80gb. You're probably not going to be able to abliterate it either, because it's moe. Other options use more vram, and where you'd have a fair amount of buffer with qwen, you're pressed with other big models.
Honestly two rtx 8000s would probably have a better return on investment than the red v2. I have an eight gpu server, five rtx 8000, three rtx 6000 ada. For basic inference, the 8000s aren't bad at all. I'm sure the green with four rtx pro 6000s are dramatically faster, but there's a $25k markup I don't honestly understand.
What models are you testing? A 120b model with hybrid attention should fit within 80gb of VRAM fine at a 4-bit quant. Also, 4-bit quants that are done well are generally fine. They certainly don’t make the model unusable.
There's some irony in the fact that this website reads as extremely NOT AI-generated, very human in the way it's designed and the tone of its writing.
Still, this is a great idea, and one I hope takes off. I think there's a good argument that the future of AI is in locally-trained models for everyone, rather than relying on a big company's own model.
One thought: The ability to conveniently get this onto a 240v circuit would be nice. Having to find two different 120v circuits to plug this into will be a pain for many folks.
I find that the most respected writing about AI has very few signs of being written by AI. I'm guessing that's because people in the space are very sensitive to the signs and signal vs. noise.
And because people writing anything worth reading are using the process of writing to form a proper argument and develop their ideas. It’s just not possible to do that by delegating even a small chunk of the work to AI.
I found it useful to preface with
* this section written by me typing on keyboard *
* this section produced by AI *
And usually both exist in document and lengthy communications. This gets what I wanted across with exactly my intention and then I can attach 10x length worth of AI appendix that would be helpful indexing and references.
> attach 10x length worth of AI appendix that would be helpful indexing and references.
Are references helpful when they're generated? The reader could've generated them themselves. References would be helpful if they were personal references of stuff you actually read and curated. The value then would be getting your taste. References from an AI may well be good-looking nonsense.
I agree wholeheartedly, I don’t see any balance in the effort someone dedicated to generating text vs me consuming it. If you feel there’s further insight to be gained by an llm, give me the prompt, not the output. Any communication channel reflects a balance of information content flowing and we are still adjusting to the proper etiquette.
"The user could have written the code themselves"
Yes, sometimes this is true, but not always.
Note, it's not one prompt (there aren't really "one prompt" any more, prompt engineering is such a 2023-2024 thing), or purely unreviewed output. It's curated output that was created by AI but iterated with me since it goes with and has to match my intention. And most of the time I don't directly prompt the agent any more. I go through a layer of agent management that inject more context into the agent that actually work on it.
If you’re spending $65,000 on this thing, needing two circuits seems like a minor problem
they could had gone with the Max-Q version RTX PRO 6000 and only require 120V circuit. 10% performance hit, but half the power.
fundamentally, looks like they are shipping consumer off-the-shelf hardwares in a custom box.
Yeah, the other big benefit is that the Max-Q's have blowers that exhaust the hot air out of the box, the workstation cards would each blow their exhaust straight into the intake of the card behind it. The last card in that chain would be cooking, as the air has already been heated up by 1800W, essentially a hair dryer on high.
Or could be the server edition 6000s that just have a heatsink and rely on the case to drive air through them, those are 600W cards.
The $12,000 one also requires it.
The specs show that it only has one PSU. The docs just say that it has 2 and thus needs two circuits, but I’d guess that was meant to be for the more expensive one.
Easier to get two circuits than rewire a breaker in an office you might be renting, no?
(I work for an electrical contractor so my sense of ease might be overcorrecting)
And 240v is orders of magnitude more common worldwide than 120v
Surprisingly affordable but I’m not really interested in the 9070XT.
If it shipped with like 4090+ (for a higher price) it’d be more tempting.
They offered a version a few months ago with 4x5090 for 25k
https://x.com/__tinygrad__/status/1983917797781426511
Stopped due to raising GPU prices:
https://x.com/__tinygrad__/status/2011263292753526978
9070XT provide roughly same inference performance at double the power, half the cost, as RTX PRO 4500. So this one is optimized for total BOM cost.
Good? That's what I want out of all websites. I don't want to read what an AI believes is the best thing for a website, I want to know the honest truth.
I don't view this as irony. This seems like good sense in understanding when AI usage will make things better and when it will not.
I am a little surprised that they openly solicit code contributions with "Invest with your PRs" but don't have any statement on AI contributions.
Maybe the volume for them is ok that well-intentioned but poor quality PRs can be politely(or otherwise, culture depending) disregarded and the method of generation is not important.
Tinygrad sure shared a few opinions on AI PRs on Twitter. I believe the gist was "we have Claude code as well, if that's all you bring don't bother".
That's a pretty excellent take, IMO. Just an undirected AI model doesn't do much, especially when the core team has time with the code, domain expertise, _and_ Claude.
I'm starting to think that if you have an AI repo thats basically about codegen, you should just close all issues automatically, the manually (or whatever) open the ones you/maintainers actually care about. Thats about the only way to kill some of the signal/noise ratio AIs are creating.
Then you could focus fire, like the script kiddies did with DDoS in the old days on fixing whatever preferred issues you have.
"locally-trained models for everyone"
Wouldn't there be a massive duplication of effort in that case? It'll be interesting to see how the costs play out. There are security benefits to think about as well in keeping things local-first.
There are multiple efforts for 'folding at home' but for AI models at this point. I get the impression that we will see a frontier model released this year built on a system like this.
3200W at ~240V is ~15A, that's just a regular household socket, at least in Europe. I imagine 240V sockets in the US are at least 15A.
No need for separate circuits, just use a double adapter.
If I'm spending at least 12k USD on the machine then doing some electrical works to accommodate it is not a big deal.
When you’re dealing with this kind of power it’s easier just to colocate where you’ll typically get two separate feeds of 208v
A typical U.S. 240V circuit is actually just two 120V circuits. Fairly trivial to rewire for that.
It's more accurate to say that the typical 120V circuit is just a 240V source with the neutral tapped into the midpoint of the transformer winding.
This. It definitely comes in at a higher voltage.
Sort of? It’s 120V RMS to ground.
yes, this is accurate for US and “works” but it’s against code here. you’ll get mildly shocked by metallic cabinets and fixtures especially if you’re barefoot and become the new shortest path to ground.
old construction in the US sometimes did this intentionally (so old, the house didn’t have grounds. Or to “pass” an inspection and sell a place) but if a licensed electrician sees this they have to fix it.
I’m dealing with a 75 year old house that’s set up this way, the primary issue this is causing is that a 50amp circuit for their HVACs are taking a shorter path to ground inside the house instead of in the panel.
As a result the 50 amp circuit has blown through several of the common 20amp grounds and neutrals and left them with dead light fixtures and outlets because they’re bridged all over the place.
If an HVAC or two does this, I’d advise against this for your 3200 watt AI rig.
EU, you don’t want to try to energize your ground. They use step down transformers or power supplies capable of taking 115-250 (their systems are 240-250V across the load and neutral lines. Not 120 across the load and neutral like ours.)
in the US. you’re talking about energizing your ground plane with 120v and I don’t want to call that safe… but it’s REALLY NOT SAFE to make yourself the shortest path to ground on say. a wet bathroom floor. with 220V-250v.
> I’m dealing with a 75 year old house that’s set up this way
I can’t tell what practice you’re referring to. Are you perhaps referring to older wiring that connects large appliances to a neutral and two hots but no ground, e.g. NEMA 10-30R receptacles? Those indeed suck and are rather dangerous. Extra dangerous if the neutral wiring is failing or undersized anywhere.
But even NEMA 10-30R receptacles are still 120V RMS phase-to-ground. (And, bizarrely, there’s an entire generation of buildings where you might find proper 4-conductor wiring to the dryer outlet and a 10-30R installed — you can test the wiring and switch to 14-30R without any rewiring.)
The exception for residential wiring is when the neutral feed from the utility transformer fails, in which case you may have 240V phase-to-phase with the actual Earth floating somewhere in the middle (via the service’s ground connection), which can result in phase-to-neutral and phase-to-ground measured anywhere in the house varying from 0 to 240V RMS.
> wet bathroom floor
A GFCI receptacle adds a considerable degree of safety and can be installed with arbitrarily old wiring. It’s even permitted by code to install one with no ground connection as long as you label it appropriately — look it up in your local code.
it’s worse than no ground connection. There’s no neutral connection so they replaced neutrals with ground.
I believe that’s kinda naughty.
It works, but it energizes your ground plane and people do get mildly shocked. that’s making me a little nervous.
So holes have been drilled in ceilings and walls and single wire neutrals or grounds have been fished down the walls, repeatedly, by yours truly , but there’s still at least one “gfci” outlet that’s wired this way And they’re balking at getting an electrician back out here for.
bridging neutral to ground because the neutral lines dead, uh, “works” to be technical but whoever did this moved on years ago and heaven only knows how many outlets or fixtures this was done in. I’m just finding out one by one as someone goes “hey this stopped working!”and you pull it and the neutral or ground blew like a fuse.
So that’s my whole point, this is an extremely bad idea for a 3200watt computer.
yes, they are all getting snipped and blank wall plated and marked as hazards that need to be remediated with a Dymo labeler as I discover them.
I don’t work here I just live here and have kind of a slummy owner who doesn’t want to do anything about any of it and doesn’t care if the plumbing or electrical works.
But they paid some guy like $4000 to install a totally unnecessary subpanel that’s bridging conflicting phases into the same circuits because he didn’t figure out this was what was going on. Dios Mio. I would have fixed the whole house for $1000. Miracle this hovel hasn’t burned to the ground yet.
I’m putting up with it for now but should probably bail before it does.
Late reply: I think you misunderstood my comment. I was replying to:
> It definitely comes in at a higher voltage.
The voltage supplied to a US house is 120V RMS measured phase-to-ground. You will not find a higher voltage in your house. This does not mean that it’s appropriate to run any non-negligible current from phase to the ground (green / “equipment grounding conductor”) wires.
One can get vaguely close to an accurate understanding by imagining that there are four wires coming out of your main panel: +120V, -120V, 0V white (the “actually use me” wire) and 0V green (a safety wire where any current more than a few mA or maybe tens of mA depending on application is at least a mistake). There’s no 240V to be found.
This explanation falls apart pretty quickly — the US system is AC, not DC.
If you actually use two 120V circuits that way and one breaker flips the other half will send 120V through the load back into the other circuit. So while that circuit's breaker is flipped it is still live. Very bad. Much better to use a 240V breaker that picks up two rails in the panel.
I assume the device has two separate PSUs, each of which accepts 120-240V, and neither of which will backfeed its supply.
They make connected circuit breakers for this use case, where one tripping automatically trips both.
i am guessing, without any proof, that, when one breaker fails the server lose it all, or loose two GPUs, depending on whether one connected to the cpu side failed.
GPUs aren't electrically isolated from the motherboard though. An entire computer is a single unified power domain.
The only place where there's isolation is stuff like USB ports to avoid dangerous ground loop currents.
That said I believe the PSU itself provides full isolation and won't backfeed so using two on separate circuits should (maybe?) be safe. Although if one circuit tripped the other PSU would immediately be way over capacity. Hopefully that doesn't cause an extended brownout before the second one disables itself.
Yes, if you have a 240V US split-phase circuit you could make a little sub-panel with a 40A breaker feeding two 20A 120V circuits and plug the two power supplies into each side. (1600W would need a 20-A breaker because 13.3A would be too much of a 15A circuit). But it would probably make more sense to just plug them both into the same 40A 240V circuit. If you use NEMA 6-20, make sure you label it appropriately and probably color it red.
In Europe, you could plug the two power supplies into an appropriately sized 240V circuit.
In an apartment you can't rewire, you could set it up in your kitchen, which in the modern US code should have two separate 20A circuits. You will need to put it to sleep while you use appliances.
A US circuit is.
But this is re: European 240/250 which is 240 between its load and neutral
I’d say don’t energize either systems ground plane, but , really, don’t do this in EU
I think you're forgetting the wires? If you have one outlet with a 15-20A 120V circuit, then the wiring is almost certainly rated for 15-20A. If you just "combined" two 120V circuits into a 240V circuit, you still need an outlet that is rated for 30A, the wires leading to it also need to be rated for 30A, and it definitely needs a neutral. So you still need a new wire run if you don't have two 120V circuits right where you wanna plug in the box. To pass code you also may need to upsize conduit. If load is continuously near peak, it should be 50A instead of 30.
So basically you need a brand new circuit run if you don't have two 120V circuits next to each other. But if you're spending $65k on a single machine, an extra grand for an electrician to run conduit should be peanuts. While you're at it I would def add a whole-home GFCI, lightning/EMI arrestor, and a UPS at the outlet, so one big shock doesn't send $65k down the toilet.
Correct me if I’m wrong, but doubling the volts doesn't change the amps, it doubles the watts. Watts = V*A.
Yes; I assumed 30A was minimum requirement for 240V service in US. Apparently I was wrong, 20A 240V is apparently normal. So in theory you could use a pre-existing 20A 120V circuit's wiring for a 240V (assuming it was 12/3 cable). And apparently 4-wire is now the standard for 240V service in US? Jesus we have a weird grid.
Doubling the volts halves the amps. P = I * V indeed.
I think you might've misread GP. (Or maybe I did?)
He's not saying you would use it as two separate 120v circuits sharing a ground but rather as a single 240v circuit. His point is that it's easy to rewire for 240v since it's the same as all the other wiring in your house just with both poles exposed.
Of course you do have to run a new wire rather than repurpose what's already in the wall since you need the entire circuit to yourself. So I think it's not as trivial as he's making out.
But then at that wattage you'll also want to punch an exhaust fan in for waste heat so it's not like you won't already be making some modifications.
The wiring (at least in the US) to the 120V outlets is just one half of the split-phase 240V. If you want to send 240V down a particular wire, you can do that, by changing the breaker, but then you lose the neutral. You also make the wires dangerous to people who don't realize that the white wire is now energized at 120V over ground. (Though it's best to test to be sure anyway, as polarity gets reversed by accident, etc.) Live wires should be black or red.
I’ve actually had half of my dryer outlet fail when half of the breaker failed.
Can confirm.
Sometimes. 240V circuits may or may not have a neutral.
Why is hn so obsessed Scott whether something is _written_ by ai or not? Who cares? Judge content, not form.
Oh wait, I get it, it's bike shedding.
I've been seeing variations on your comment a lot on HN lately and I find it a rather vapid way of looking at something so intricate as human communication. Among other things, the medium is the message!
Big companies are pushing cloud really hard, and yea the hardware prices too is a problem. People still buy Google cloud and OneDrive when they could literally pickup an old computer from trash and Frankenstein it into a NAS server.
The exabox is interesting. I wonder who the customer is; after watching the Vera Rubin launch, I cannot imagine deciding I wanted to compete with NVIDIA for hyperscale business right now. Maybe it’s aiming at a value-conscious buyer? Maybe it’s a sensible buy for a (relatively) cash-strapped ML startup; actually I just checked prices, and it looks like Vera Rubin costs half for a similar amount of GPU RAM. I’m certain that the interconnect will not be as good as NV’s.
I have no idea who would buy this. Maybe if you think Vera Rubin is three years out? But NV ships, man, they are shipping.
Sometimes you can compete with the big boys simply because they built their infra 5+ years ago and it’s not economically viable for them to upgrade yet, because it’s a multi-billion dollar process for them. They can run a deficit to run you out of the business, but if you’re taking less than 0.01% of their business, I doubt they’d give a crap.
Have to imagine each tinybox is targeting different tiers of startups trying to fine-tune/RL their way to custom models for narrow use-cases.
Maybe the target profile for exabox looks like a smaller/younger Cursor? If you're a small team with some seed funding and expertise, this kind of compute in a single box you can set up in your office feels like it could be a great fit.
> The exabox is interesting.
Can it run Crysis?
Yes, it can generate Crysis with diffusion models at 60 fps.
Only gamers understand that reference
-- Jensen Huang
*Only gamers know that joke.
I’m not a gamer but I know that joke
Probably, the rdna5 can do graphics. But it would be a huge waste, since you could probably only use one of the 720 GPUs
The problem with all these "AI box" startups is that the product is too expensive for hobbyists, and companies that need to run workloads at scale can always build their own servers and racks and save on the markup (which is substantial). Unless someone can figure out how to get cheaper GPUs & RAM there is really no margin left to squeeze out.
Would a hedge fund that does not want to trust to a public AI cloud just buy chassis, mobos, GPUs, etc, and build an equivalent themselves? I suspect they value their time differently.
Why do you think a hedge fund can't hire a couple of IT guys? Most of the larger ones have technical operations that would put big tech to shame.
Medium sized hedge funds are a good portion of the market, and only really want to hire just enough tech people to keep the quant pipelines running.
[dead]
They wouldn’t build anything - they would order from Dell or Supermicro.
We may be surprised how illefficient companies are in organizing the creation of sophisticated things (including processes) for themselves, to use (so for the cost center column).
Higher management figures out things to do in strategic level, in brief, and pushes on "soldiers", who kick it through in the least time (cheapest of the cheapest, for the sake of the quarterlies) EXACTLY the way management told it. Because they have to, their job is to make happen the company objectives given, the way it is given. Pushing out crap in the shape of the thing expected.
Larger organiztaion can use these kind of things the most. Even if they don't do that.
They’re kickstarting a TINY device that is pocketable and aimed at consumers. I’ve backed it (full disclosure).
https://www.kickstarter.com/projects/tiinyai/tiiny-ai-pocket...
This is not the same company. The OP Tiny Corp accused them of Trademark infringement on Twitter, due to exactly this kind of misconception.
I might’ve been conned: https://bay41.com/posts/tiiny-ai-pocket-lab-review/
[dead]
[flagged]
just want to run a 7-8b model locally
This is already solved by running LM Studio on a normal computer.
Ollama or llama.cpp are also common alternatives. But a 8B model isn't going to have much real-world knowledge or be highly reliable for agentic workloads, so it makes sense that people will want more than that.
the compute density is insane. but giving a 70B model actual write access locally for agentic workloads is a massive liability. they still hallucinate too much. raw compute without strict state control is basically just a blast radius waiting to happen.
$12,000 for the base model is insane. I have an Apple M3 Max with 128GB RAM that can run 120B parameter models using like 80 watts of electricity at about 15-20 tokens/sec. It's not amazing for 120B parameter models but it's also not 12 grand.
M3 max tflops is tiny compared to the 12k box. It's not even comparable.
It is very comparable if you work out the $/tok/s on inference. I did some napkin math and it looks like you’re getting roughly 3x the performance for 3x the cost. Red v2 vs Mac Studio M3 Ultra 96GB.
If you compare tokens/kWh efficiency then my math has Mac Studio being about 1.5x more efficient.
M3 has tolerable decode performance for the price, and that's what people would care about most of the time. they underperform severely wrt. prefill, but that's a fraction of the workload. AI, even agentic AI, spends most of its time outputing tokens, not processing context in bulk.
it's for fools. i bought 160gb of vram for $1000 last year. 96gb of p40 VRAM can be had for under $1000. And it will run gpt-oss-120b Q8 at probably 30tk/sec
P40 is Tesla architecture which is no longer receiving driver or CUDA updates. And only available as used hardware. Fine for hobbyists, startups, and home labs, but there is likely a growing market of businesses too large to depend on used gear from ebay, but too small for a full rack solution from Nvidia. Seems like that's who they're targeting.
99% of interest is in inference. If you want to fine-tune a model, just rent the best gpu in the cloud. It's often cheaper and faster.
Great option if you don't mind sharing your data with the cloud. Some businesses want to own the hardware their data resides on.
How many businesses have the capabilities and expertise to train their own models?
No idea. Probably more every day.
renting GPU, how is that sharing data with the cloud? you can rent GPU from GCP or AWS
I suppose if I rent a cloud GPU and just let it sit there dark and do nothing then I wouldn't have to move any data to it. Otherwise, I'm uploading some kind of work for it to do. And that usually involves some data to operate on. Even if it's just prompts.
So you also believe when you rent a server you are sharing your data with the cloud? AWS and GCP are copying all private data on servers? Give me a break. There's a big difference between renting a server and using an API.
> So you also believe when you rent a server you are sharing your data with the cloud [hosting provider]?
Only if you upload your data to that cloud server you rented. Then, by definition, you are.
> AWS and GCP are copying all private data on servers?
Every computer copies data when moving it. Several times, in fact. Through network card buffers, switches, system memory, disk caches, and finally to some form of semi-permanent storage.
I don't have to think Amazon is stealing my data to be aware that Amazon S3 buckets containing privileged information are routinely found open. I don't have to think that Google is spying on me to know that operating equipment my business owns on prem and does not share requires me to trust fewer people and less complex systems than doing the same work from the cloud.
You are very quick to make foolish assumptions and assign them to others.
> In order to keep prices low and quality high, we don't offer any customization to the box or ordering process. If you aren't capable of ordering through the website, I'm sorry but we won't be able to help.
Has this guy never worked on a B2B product before? Nobody is going to order a $10 million piece of infrastructure through your website's order form. And they are definitely going to want to negotiate something, even if it's just a warranty. And you'll do it because they're waving a $10 million check in your face.
The tone of this website is arrogant to the point of being almost hostile. The guy behind this seems to think that his name carries enough weight to dictate terms like this, among other things like requiring candidates to have already contributed to his product to even be considered for a job. I would be extremely surprised if anyone except him thinks he's that important.
I haven’t seen tinygrad used for any mainstream production project or thing of value, yet.
Besides a lot of self congratulatory pats on the back for how elegant it is. Honestly, when I read it, it looked confusing as all the other ML libraries. Not actually simple like Karpathy’s stuff.
All that to say, I do really want it to succeed. They should probably hire some practical engineers and not just guys and gals congratulating themselves how elegant and awesome they are.
Your framing of this section is misleading. On the site it's preceded by a FAQ-style 'question':
> Can you fill out this supplier onboarding form?
That's very important context, as anyone who has been asked to fill out a supplier onboarding form (hi) will attest.
Filling out an onboarding form is an example of what he's not willing to do, not the only thing he isn't willing to do.
> we don't offer any customization to the box or ordering process
Every B2B deal of that size that I've ever seen requires at least weeks of meetings between the customer and vendor, in which every detail is at least discussed if not negotiated. That would certainly constitute a "customization" to this guy's prescribed ordering process, which is to "Buy it now" [1] through the website at the stated price like you're ordering a jar of peanuts on Amazon. This is not "framing", it's what the guy said. If it isn't what he meant then he needs to fix his copy.
[1] Yes, there is an actual "Buy it now" button for a $65,000 business purchase that takes you to a page that looks just like a Stripe form. There isn't even a textbox for delivery instructions. Wild.
Then if they succeed, I guess you're going to see a different process for the first time in your life.
On a website where we frequently talk about disruptive business models, this whole attitude kinda stinks.
> Then if they succeed, I guess you're going to see a different process for the first time in your life.
Sure, I guess. Far more likely that they won't succeed, and it will be because of their pointless refusal to cooperate with others. I'm curious why you think we should "disrupt" companies putting a little due diligence into massive purchases.
> On a website where we frequently talk about disruptive business models, this whole attitude kinda stinks.
I could say the same thing about making a comment like this on a website where groupthink is rightfully mocked.
> you're going to see a different process for the first time in your life
That sounds very neutral, but wouldn't this, by removing the human element and flexibility from business transactions, be a further step along a general enshittification trend?
> arrogant to the point of being almost hostile
First encounter with geohot eh?
What does this mean? Is it some reference to different temperaments across geographies? Or some Internet slang?
https://en.wikipedia.org/wiki/George_Hotz
He's not actually selling the exabox yet. It sounds like he put up a hypothetical config to see if anyone is interested.
There isn't a $10MM device right now, just $64M and under. I doubt the order process will remain the same in 12 months when the $10MM device becomes available
The specs for the “exabox” scream “this is a joke” to me.
> 20,000 lbs
> concrete slab
Huge-scale IT systems are typically delivered in one or more 42/44u cabinets, and are designed to be installed on raised floors.
It's a shipping container. Look at the dimensions. They say concrete slab probably half as a joke, half because building code would require it to consider it a non-temporary structure.
It's a shipping container that you install outdoors.
Are you referring to the images of branded shipping containers on their Twitter page that have visible Gemini watermarks … and jokes in the comments about AI trailer parks?
20x8x8.5 ft is the dimensions of a half shipping container. You think that render is a joke but it's not. They don't have photos yet because it's a 2027 product (if it actually comes out which I would bet against).
It's also funny that they explicitly list driver quality as "good" for the base option and "great" for the intermediate one. You're really going to deliberately provide worse drivers for the machine I paid you for, just because I didn't buy the more expensive one?
I mean I'm sure lots of companies do this in practice because tickets for higher-paying customers naturally get prioritized, but directly stating your intention to do it on your home page is hilarious.
Nvidia drivers are better than AMD. It's not really something they have control over. Geohot is definitely obsessed with bitching about driver bugs though.
That may be, but then it's an inside joke that many of his customers won't get. It just looks like a "fuck you" to anyone buying the cheaper system.
This guy desperately needs a marketing intern to look over his copy. Or hell, anyone who knows how to talk to humans.
Not a joke. It's just true.
It doesn't matter if it's a joke. The non-technical manager or VP making this purchase will not understand it and will expect poor treatment from this vendor, an expectation that will be reinforced by numerous other things on this page. There is no reason to include it at all.
It doesn’t read as if they actually care about broad appeal, given their plain refusal to accommodate traditional procurement processes
So they're only interested in taking on customers who are OK with being treated poorly?
Anyone who can’t deal with their procurement process isn’t a customer.
There’s nothing remotely unusual about being selective about who or how to bring on new customer in B2B sales.
Their preference is for more simplicity than normal —- many businesses make it much harder
To me it signals honesty. But this is a subjective judgement. It really sounds like you subjectively disliked the page, and you're trying to present that dislike as objective fact. It really annoyed me the way you kept changing your argument to justify that. Why not just say "I dislike their marketing copy, it rubbed me the wrong way" and leave it at that?
I'm confused by this comment. Is it not obvious that everything I've said is my opinion?
Not everyone feels the need to hedge everything they say with "to me..." and "it really sounds like...".
> It really annoyed me ...
I have no idea what you expect me to do with that information.
> you kept changing your argument
I'm not sure what you're referring to. Are you talking about the joke/not-a-joke thing? I didn't change my argument, I dismissed a shallow objection to an irrelevant detail. My point is that regardless of why the driver quality was included on the page, customers are going to take it the wrong way. Yes, that is my opinion, because apparently that needs to be explicitly stated.
Everyone else who disagreed with me seems to have understood all this so I don't know what the source of your confusion is.
It seems that you work a lot with managers who have no clue what they are buying and why.
I mean, you're not wrong: buying enterprise software from Oracle or Microsoft or Salesforce is pure pain.
But nobody expects buying niche hardware from a tiny vendor to involve the usual 128 pre/post sale meetings and 256 hours of professional services.
Also, relevant VP buying these things usually do understand the difference between AMD and Nvidia stacks really well. Like, really-really well.
> It seems that you work a lot with managers who have no clue what they are buying and why.
There are certain quirks of this platform's user base that always make me laugh. For example, HNers absolutely love to imply something condescending about the other guy's workplace in order to make their point.
Watch this, I can do it too: Working with managers who make $65,000 (or $10 million) purchases with no more due diligence than reading a marketing page and clicking "Buy it now" is not the flex you think it is.
I was involved in it-related deals on both purchasing and selling sides. Sums involved were larger than both numbers you mentioned.
And I honestly see almost no correlation between the amount of negotiation involved, and value received.
Some of the most useful things we've integrated were either free or meant that only the "buy it now" button had to be clicked.
Some of the absolutely worst systems I had to work with were purchased after making a call to that "let us know" number.
This tiny guy is mostly saying that he doesnt have the time for enterprise bla-bla. I am not sure he can organise enterprise sales with this attitude but can definitely relate to it!
I took that as a dig against AMD vs Nvidia driver quality.
I guess it is called ‘honesty’.
I imagine that the FAQ might get updated when there’s actually a $10M machine for sale
Maybe. Frankly I'd be very surprised if any business ordered a $65k machine that way either.
Yeah it’s a little odd. Maybe they are meant to be really really cool toys? People regularly spend more than $65k on things like cars to show off, so it could be like that.
I have no use for these but I might buy one anyway if I won the lottery. ¯\_(ツ)_/¯
> arrogant to the point of being almost hostile.
The YouTube rap video of geohotz telling Sony lawyers suing him to blow him is still up.
His style of dealing with corporate matters is certainly unconventional
Well, at least he had the power that average joes don't have. And he used it well.
Tinybox is cool but I think the market is maybe looking more for a turn-key explicit promise of some level of intelligence @ a certain Tok/s like "Kimi 2.5 at 50Tok/s".
Is this like the new equivalent of crypto mining? I remember the early days when they would sell hardware for farming crypto, now it’s AI?
Kind of yes, except there is no block reward.
The block reward is firing humans and collecting ad revenue for slop
Where is the 120B documented? This seems to be an editorialized title.
Edit: found a third party referencing the claim but it doesn't belong in the title here I think:
Meet the World’s Smallest ‘Supercomputer’ from Tiiny AI; A Machine Bold Enough to Run 120B AI Models Right in the Palm of Your Hand
https://wccftech.com/meet-the-worlds-smallest-supercomputer-...
That third party link is from a different company (Tiiny with an extra i)
Now I'm wondering if the HN title was submitted by some AI bot that couldn't tell the difference.
Ha, good catch, I googled for Tinybox 120B and clearly didn't read the article beyond the seeming match.
IDK, I feel it’s quite overpriced, even with the current component prices.
I almost sure it’s possible to custom build a machine as powerful as their red v2 within 9k budget. And have a lot of fun along the way.
AMD now has 32 GiB Radeon AI Pro 9700. 4 of these (just under 2k each) would put you at 128 GiB VRAM
VRAM is not everything - GPU cores also matter (a lot) for inference
inference speed is like monitor Hz; sure, you go from 60 to 120Hz and thats noticeable, but unless your model is AGI, at some point you're just generating more code than you'll ever realistically be able to control, audit and rely on.
So, context is probably more $/programming worth than inference speed.
4x Radeon will have significantly more GPU power than say Mac Studio or DGX Spark.
Perhaps this company should think about acting as a landlord for their hardware. You buy (or lease) but they also offer colocation hosting. They could partner with crypto miners who are transitioning to AI factories to find the space and power to do this. I wonder if the machines require added cooling, though, in what would otherwise be a crypto mining center. CoreWeave made the transition and also do colocation. The switchover is real.
I think Tinygrad should think about recycling. Are they planning ahead in this regard? Is anyone? My thought is if there was a central database of who own what and where, at least when the recycling tech become available, people will know where to source their specific trash (and even pay for it.) Having a database like that in the first place could even fuel the industry.
The incremental price increases between products is funny.
$12,000, $65,000, $10,000,000.
I was more worried by the 600kW power requirement... that's 200 houses at full load (3kw) in southern europe... which likely means 400 houses at half load.
the town near my hometown has 650 – 800 houses (according to chatgpt).
crazy.
Or it's two 300kW fast EV chargers working together.
A typical home just consumes rather little energy, now that LED lighting and heat pump cooling / heating became the norm.
I think the above commentor is reflecting on the total energy use from having a 600KW load running 24/7. I suppose the more interesting observation is the 14 MWh of daily consumption, enough to charge 100 Rivians every day.
> and heat pump cooling / heating became the norm.
We're not all solidly middle-class (especially in Southern and Eastern Europe) and as such we cannot afford those heat pumps. But we'll have to eat the increased energy costs brought by insane server configurations like the ones from the article, so, yeey!!!
> now that LED lighting and heat pump cooling / heating became the norm.
My brother in Christ, you vastly overestimate southern europe
I noticed that Southern Europe often basically ignores both heating and cooling, especially close to the warm sea.
But with heat pumps becoming normal in the North and in the US, they become mass-produced, and the prices fall. Same has happened to LED lamps.
> at full load (3kw)
Do you live in a deprived rural village in a very poor country? Because you can't even run a heater and the oven with 3kW.
No it’s quite the norm actually.
Most power contracts give you 3 kwh power supply for residential home. That’s the standard.
Bumping to 4.5 or 6kwh must be required explicitly and costs and extra on the base power supply bill
Your hometown also has public lightning, water pumps, and probably some other stuff.
That’s surprising, 200 amp 240v service is pretty common in the US.
I mean the difference in performance is quite big too. However, the 10,000,000 is a little bit too much (imo).
https://en.wikipedia.org/wiki/Decoy_effect
I would love to see real-life tokens/sec values advertised for one or various specific open source models.
I'm currently shopping for offline hardware and it is very hard to estimate the performance I will get before dropping $12K, and would love to have a baseline that I can at least always get e.g. 40 tok/s running GPT-OSS-120B using Ollama on Ubuntu out of the box.
Look for llmfit on github. This will help with that analysis. I've found it reasonably accurate. If you have Ollama already installed, it can download the relevant models directly.
For reference, 12k gets you at least 4 Strix Halo boxes each running GPT-OSS-120B at ~50tok/s.
Not sure why they stopped using 6 GPUs in thei builds - with 4 GPUs, both 9070 and rtx6000 come in 2 slot designs, so it easy to build it yourself using a bit more expensive, but still fairly regular motherboard.
With 6 GPUs you have to deal with risers, pcie retimers, dual PSUs and custom case for so value proposition there was much better IMO
Sound like solid prebuilt with well balanced components and a pretty case
Not revolutionary in any way, but nice. Unless I'm missing something here?
It's pretty close to what people have been frankenbuilding on r/LocaLLaMa... It's nice to have a prebuild option.
You could also order such configurations from a classic server reseller as far as I know. The case is a bit original there.
Tiny boxes are already several years old IIRC
If you wanted a box built by geohot, most recently known for signing on to Elons Twitter and then bailing, it's for you
actually known for comma.ai
Cool that you have a dual power supply model. It says rack mountable or free standing. Does that mean two form factors? $65K is more than we can afford right now but we are definitely eventually in the market for something we can run in our own colo.
It's funny though... we're using deepseek now for features in our service and based on our customer-type we thought that they would be completely against sending their data to a third-party. We thought we'd have to do everything locally. But they seem ok with deepseek which is practically free. And the few customers that still worry about privacy may not justify such a high price point.
Most privacy talk folds on contact with a quote. Latency and convenience beat philosophy fast once someone wants a dashboard next week, and a lot of "data sensitivity" talk is just the corporate version of buying "organic" food until the price tag shows up.
If private inference is actually non-negotiable, then sure, put GPUs in your colo and enjoy the infra pain, vendor weirdness, and the meeting where finance learns what those power numbers meant.
The real case for private inference is not "organic", it's "slow food". Offering slow-but-cheap inference is an afterthought for the big model providers, e.g. OpenRouter doesn't support it, not even as a way of redirecting to existing "batched inference" offerings. This is a natural opening for local AI.
But how slow is too slow (faster than you’d think) and even then, you’re in for $25,000 for even the most basic on-premise slow LLM.
[dead]
"... and likely the best performance/$".
"likely" doesn't inspire much confidence. Surely, they have those numbers, and if it was, they'd publicize the comparisons.
Regarding 2x faster than pytorch being a condition for tinygrad to come out of alpha:
Can they/someone else give more details as to what workloads pytorch is more than 2x slower than the hardware provides? Most of the papers use standard components and I assume pytorch is already pretty performant at implementing them at 50+% of extractable performance from typical GPUs.
If they mean more esoteric stuff that requires writing custom kernels to get good performance out of the chips, then that's a different issue.
$12,000 gets you 1Gb/s networking and vanilla Ubuntu 24.04. Napkin math on the hardware it looks like margins are around 50% which feels like a school fundraiser where everyone pays what is obviously way more than normal retail price for X because "it's for the children."
I'm not sure what tinygrad is but I assume the markup is because the customer is making a conscious choice to support the tinygrad project. But what's unusual is there is apparently no reason whatsoever to buy this hardware, even if you plan on using tinygrad exclusively for your project. At least with System76 hardware I get (in theory) first class support for Pop!_OS.
Curious to know who will spend this much money without external funding? Would you spend any VC invested money into this nameless brand? Are there any guardrails or clauses to protect the kind of expenses?
What’s the most effective ~$5k setup today? Interested in what people are actually running.
Depends. If token speed isn't a big deal, then I think strix halo boxes are the meta right now, or Mac studios. If you need speed, I think most people wind up with something like a gaming PC with a couple 3090 or 4090s in it. Depending on the kinds of models you run (sparse moe or other), one or the other may work better.
At $7.2k + tax:
* RAM - $1500 - Crucial Pro 128GB Kit (2x64GB) DDR5 RAM, 5600MHz CP2K64G56C46U5, up to 4 sticks for 128GB or 256GB, Amazon
* GPU - $4700 - RTX Pro 5000 48GB, Microcenter
* CPU/Mobo bundle - $1100 - AMD Ryzen 7 9800X3D, MSI X870E-P Pro, ditch the 32GB RAM, Microcenter
* Case - $220, Hyte Y70, Microcenter
* Cooler - $155, Arctic Cooling Liquid Freezer III Pro, top-mount it, Microcenter
* PSU - $180, RM1000x, Microcenter
* SSD - $400 - Samsung 990 pRO 2TB gen 4 NVMe M.2
* Fans - $100 - 6x 120mm fans, 1x 140mm fan, of your choice
Look into models like Qwen 3.5
> RAM - $1500 - Crucial Pro 128GB Kit (2x64GB) DDR5 RAM
I knew prices went up, but that's wild. I bought 64GB (2x32) of RAM a year ago for $90.
Surprised to see X3D given the reports of failures. I’ve opted for a regular 9900x and X670E-E just to have a bit more assurance.
$7.2k just to run at best Qwen3.5-35B-A3B doesn't seem worth it at all.
This is certainly not the most effective use of $7k for running local LLMs.
The answer is a 16" M5 Max 128GB for $5k. You can run much bigger models than your setup while being an awesome portable machine for everything else.
Performance (tok/s and PP) or quality (model size)? Pick one.
In terms of GPU memory bandwidth (models fitting in the ~48GB of RTX 5000 Pro card), the RTX card I described above has over 2x the bandwidth of an M5 Max.
If leveraging system RAM (the 128GB-256GB outside the GPU) to run larger models, then the memory bandwidth is ~6x slower than M5 Max.
For models fitting in the ~48GB RTX memory, like dense Qwen3.5 27B models, the RTX will be 2-4x faster than M5 Max. For models that don't fit in the 48GB RTX memory, the M5 Max will be 5-20x faster.
Also worth considering future upgrades: Do you plan to throw away the machine in a few years, or pick up multiple used RTX 6000 Pro cards when people start ditching them?
Sadly $5k is sort of a no-man's land between "can run decent small models" and "can run SOTA local models" ($10k and above). It's basically the difference between the 128GB and 512GB Mac Studio (at least, back when it was still available).
The DGX Spark is probably the best bang for your buck at $4k. It's slower than my 4090 but 128gb of GPU-usable memory is hard to find anywhere else at that price. It being an ARM processor does make it harder to install random AI projects off of GitHub because many niche Python packages don't provide ARM builds (Claude Code usually can figure out how to get things running). But all the popular local AI tools work fine out of the box and PyTorch works great.
It's $4.7K now, darn inflation!
https://marketplace.nvidia.com/en-us/enterprise/personal-ai-...
A small joke at this weeks GTC was the "BOGOD" discount was to sell them at $4K each...
DGX Spark is a fantastic option at this price point. You get 128GB VRAM which is extremely difficult to get at this price point. Also it’s a fairly fast GPU. And stupidly fast networking - 200gbps or 400gbps mellanox if you find coin for another one.
I’m not very well versed in this domain, but I think it’s not going to be “VRAM” (GDDR) memory, but rather “unified memory”, which is essentially RAM (some flavour of DDR5 I assume). These two types of memory has vastly different bandwidth.
I’m pretty curious to see any benchmarks on inference on VRAM vs UM.
I’m using VRAM as shorthand for “memory which the AI chip can use” which I think is fairly common shorthand these days. For the spark is it unified, and has lower bandwidth than most any modern GPU. (About 300 GB/s which is comparable to an RTX 3060.)
So for an LLM inference is relatively slow because of that bandwidth, but you can load much bigger smarter models than you could on any consumer GPU.
A quick benchmark using float32 copies using torch cuda->cuda copies, comparing some random machines:
This is a "eh, it works" benchmarks, but should give you a feel for the relative performance of the different systems.In practice, this means I can get something like 55 tokens a sec running a larger model like gpt-oss-120b-Q8_0 on the DGX Spark.
Nice! Thanks for that.
55 t/s is much better than I could expect.
Internet seems to think the SW support for those is bad, and that strix halo boxes are better ROI.
Meh. DGX is Arm and CUDA. Strix is X86 and ROCm. Cuda has better support than ROCm . And x86 has better support than Arm.
Nowadays I find most things work fine on Arm. Sometimes something needs to be built from source which is genuinely annoying. But moving from CUDA to ROCm is often more like a rewrite than a recompile.
> But moving from CUDA to ROCm is often more like a rewrite than a recompile.
Isn't everyone* in this segment just using PyTorch for training, or wrappers like Ollama/vllm/llama.cpp for inference? None have a strict dependency on Cuda. PyTorch's AMD backend is solid (for supported platforms, and Strix Halo is supported).
* enthusiasts whose budget is in the $5k range. If you're vendor-locked to CUDA, Mac Mini and Strix Halo are immediately ruled out.
Most everything starts as PyTorch. (Or maybe Jax.) But the inference engines all use hand tuned CUDA kernels - at least the good ones do. You have to do that to optimize things.
I'm certain inference engines don't use hand-tuned CUDA on Radeon or Mac Mini chips. My statement holds: those engines have no strict dependency on CUDA, or they'd be Nvidia-only.
CUDA != Driver support. Driver support seems to be what's spotty with DGX, and iirc Nvidia jas only committed to updates for 2 years or something.
Can even network 4 of these together, using a pretty cheap InfiniBand switch. There is a YouTube video of a guy building and benchmarking such setup.
For 5K one can get a desktop PC with RTX 5090, that has 3x more compute, but 4x less VRAM - so depending on the workload may be a better option.
VRAM vs UM is not exactly apples to apples comparison.
Biggest Mac Studio you can get. The DGX Spark may be better for some workflows but since you're interested in price, the Mac will maintain it's value far longer than the Spark so you'll get more of your money out of it.
With $5k you have to make compromises. Which compromises you are willing to make depends on what you want to do - and so there will be different optimal setup.
Fully aware of the DGX spark I've actually been looking into AMD Ryzen AI Max+ 395/392 machines. There's some interesting things here like https://www.bee-link.com/products/beelink-gtr9-pro-amd-ryzen... and https://www.amazon.com/GMKtec-5-1GHz-LPDDR5X-8000MHz-Display... ... haven't pulled the trigger yet but apparently inferencing on these chips are not trash.
Machines with the 4xx chips are coming next month so maybe wait a week or two.
It's soldered LPDDR5X with amd strix halo ... sglang and llama.cpp can do that pretty well these days. And it's, you know, half the price and you're not locked into the Nvidia ecosystem
unfortunately the bigger models are pretty slow in token speed. The memory is just not that fast.
You can check what each model does on AMD Strix halo here:
https://kyuz0.github.io/amd-strix-halo-toolboxes/
4xx chips are less capable than the 395
> What’s the most effective ~$5k setup today?
Mac Studio or Mac Mini, depending on which gives you the highest amount of unified memory for ~$5k.
The most interesting part of Tinybox isn't just the hardware, but the push for a more vertical integration with tinygrad. We've become so accustomed to the CUDA/PyTorch stack that seeing a serious attempt at a different software-hardware synergy is refreshing, even if the hardware specs or price point relative to DIY homelabs raise some eyebrows for power users. It's more about reducing the friction for researchers who want a "just works" environment without the nightmare of driver/toolkit version hell.
Would be very curious how RL benchmarks shake out vs M5 Pro/Max.
Doubt local inference is the target use case near nearly as much as post-training. I could totally see something like this being super appealing for a startup looking to do some fine-tuning/distillation to tune a small open-weight model for a narrow use case.
It might be a bit CPU and RAM starved… Which in theory should be OK, but in practice you'll find production workloads that struggle because of this. Just make sure whatever you want to run on this is indeed extremely GPU-bound, or you might have bad surprises later.
I thought the most interesting thing about tinygrad was that theoretically you could render a model all the way into hardware similar to Taalas (tinygrad might be where Taalas got the idea for all I know).
I could swear I filed a GitHub issue asking about the plans for that but I don't see it. Anyway I think he mentioned it when explaining tinygrad at one point and I have wondered why that hasn't got more attention.
As far as boxes, I wish that there were more MI355X available for normal hourly rental. Or any.
I just don’t believe that this can run inference on a 120 billion parameter model at actually useful speeds.
Obviously any Turing machine can run any size of model, so the “120B” claim doesn’t mean much - what actually matters is speed and I just don’t believe this can be speedy enough on models that my $5000 5090-based pc is too slow for and lacks enough vram for.
Look at the GPU and RAM spec; 120b seems workable.
For the red v2?
120B could run, but I wouldn't want to be the person who had to use it for anything.
To be fair, the 120B claim doesn't appear on the webpage. I don't know where it came from, other than the person who submitted this to HN
It is more than fair, also, you're comparing your 5k devices to 12k and more importantly 65k and >10m devices.
The "to be fair" part of my comment was saying that the tinygrad website doesn't claim 120B.
Also nobody is comparing this box to an $10M nVidia rack scale deployment. They're comparing it to putting all of the same parts into their Newegg basket and putting it together themself.
Tinygrad devices are interesting, I wish I have screen captures - but their prices have gone up and some specs like RAM have gone down.
A single box with those specs without having to build/configure (the red and green) - I could see being useful if you had $ and not time to build/configure/etc yourself.
exabox reads as if it was making a joke of something or someone. if it's real then it's really interesting!
I have 8x RTX 6000 Pro. Better to run the 300 W version of the cards. And it costs close to their 4x version. I get why they make it so big. So you can cool it at home. I prefer to just put in datacenter. Much cheaper power.
Why do I get the impression that I get more bang for the buck by going through OpenRouter? Of course, not anyone can do that and there are security and other concerns.
This is cool, I'll add these as desktops to https://flopper.io!
How do you test/generate these numbers?
10 mil today... 1k in 10 years. Are OpenAI and Anthropic overvalued?
Looking at these prices I’m just thinking that as a user it makes no sense to buy this when you can just use the subsidised stuff from AI companies and then buy it a few years later at a tiny % of the cost.
Quite expensive little bastard. I wonder how much does it make sense to invest in a such device, if you can get $0.40/mtok from hyperbolic for example
If you're OK letting them train on, and maybe keep, your data, then it's hard to beat cloud prices vs. local.
exabox -
720x RDNA5 AT0 XL 25,920 GB VRAM 23,040 GB System RAM
~ $10 Million
Who is the target market here?
I can't find sources but I think they are building it for Comma.ai (geohot's other company) so that Comma can scale up their training datacenter.
And... what about 20k lbs and 1360 cubic feet screams "tiny" :)
That is very close to a half-length shipping container.
VC funded startups
A non-trivial share of this market won’t show up in public data. That makes most estimates unreliable by default
A company which doesn't want the big LLM providers to see it's prompts or data - military, health, finance, research
its a bit weird to me ud need to be contributor to their software to work in operations or hardware, but I suppose its ok for tinycompany. in long term its likely better to have domain experts and not bias everything towards the same thing.
the boxes look cool but how good are they really? the cheapest box seems pricey at 12 for a what is essentially a few gaming gpus. i dont see why you couldnt make that like half the price. u could do a PC/server build thats much much faster for way less. size doesnt matter if its more than twice the price i think...
the more expensive box has atleast real processing gpus but afaik also not very popular ones, this one seems maybe more fair priced (there seems a big difference in bang for buck between these???).
the third one suggested looks like a joke.
dont get me wrong, this seems like a really cool idea. But i dont see it taking off as the prices are corporate but the product seems more home use.
maybe in time they will find a better balance, i do respect the fact that the component market now is sour as hell and making good products with stable prices is pretty much i possible.
id love one of these machines someday, maybe when i am less poor, or when they are xD.
(love the styling of everything, this is the most critical i could be from a dumb consumer perspective, which i totally am btw.)
Oh, this is geohots product?
He's an interesting guy. Seems to be one who does things the way he thinks is right, regardless of corporate profits.
Can someone explain the exabox? They say it "functions as a single GPU". Is there anything like that currently existing?
An NVL72 rack or Helios rack also "functions as a single GPU".
TPU pods
I always wonder about these expensive products: Does the company make them once its ordered or do they just make them beforehand?
In this case, they're taking wire transfers, so they're definitely building them once they get the cash.
He builds a batch every few months.
Are we at the point where 2x 9070XT's are a viable LLM platform? (I know this has 4, just wondering for myself).
These things don’t have Flash Attention or either have a really hacked together version of it. Is it viable for a hobby? Sure. Is it viable for a serious workload with all the optimizations, CUDA, etc.. Not really.
I'd go with strix halo if you're looking at that old of tech.
the latest AMD GPUs are RX 9070 XT w/32GB each
Meanwhile M-series processors and Qwen are racing to do the same thing for a much more approachable price.
I wonder if this is frontpage right now because of the other tiiny (the names are similar) video that went viral ... which turns out wasn't an actual product by the tinygrad linked in this post[1]
[1]https://x.com/ShriKaranHanda/status/2035284883384553953
Adding this to my list of ~beautifully~ designed things to buy when I win the lottery.
Who is the intended customer for this product? I am genuinely curious.
Anyone who wants to run/train/finetune a local llm.
“Not your weights, not your brain.”
Great idea, can you publish the power consumption units for this device
Skeptical of their engineering, with replies to questions like this: https://x.com/jgarzik/status/2031312666036146460?s=20
They answered your question with a pretty specific uptime target. Calling it a dodge and then moving the goalposts with a new question as your follow up doesn’t speak to you acting in good faith.
tbh they really didn't, tinygrad's was clearly a joke response. they were not providing a real uptime target.
Can't see replies, what did they say?
https://xcancel.com/jgarzik/status/2031312666036146460?s=20
My interest in anything associated with geohot took a colossal nose dive today after seeing this post against democracy, quoting frelling M*ncius M*ldbug: Democracy is a Liability. https://news.ycombinator.com/item?id=47469543 https://geohot.github.io//blog/jekyll/update/2026/03/21/demo...
Theres a lot there that makes sense & I think needs to be considered. But a lot just seems to be out of the blue, included without connection, in my view. Feels like maybe are in-grouo messages, that I don't understand. How this is headered as against democracy is unclear to me, and revolting. I both think we must grapple with the world as it is, and this post is in that area, strongly, but to let fear be the dominant ruling emotion is one of the main definitions of conservativism, and it's use here to scare us sounds bad.
He was always defending democracy and freedom before, and that was his argument for the local AI thing? What changed?
Geohotz's politics are fairly straightforward once you understand his background. Geohotz is the prodigy child who, at the age of ~16 accomplished amazing technical feats on his own.
And his politics are a derivative of Great Man Theory, and his positions on things like democracy follow from that. This idea, and those espoused by some of the VC/tech elite like Peter Theil are that singular hardworking genius individuals can change the world on their own, and everyone who not in this top 0.1% are borderline NPCs.
They do this both because of their genius/hardwork, and also because they are willing to break the rules that are set forth by this bottom 99.9%.
I'm starting to call this ideology Authoritarian techno-Libertarianism. Its a delibriately oxymoronic name that I use, because these "Great Men" are definitely trying to change the world. IE, they are trying to impose their goals and values on the world without getting the buyin of other people.
Thats the "authoritarian" part. And then the "libertarian" part is that they are going about this imposition of their will on the world by doing it all themselves, through their own hard work.
Think "Person invents a world changing technology, that some people thing is bad, and just releases it open source for anyone to use". AI models are a great example, in fact. Once that technology is out there the genie cannot be put back into the bottle and a ton of people are going to lose their jobs, ect.
A distain for democracy follows directly from things like this. You dont wait for people to vote to allow you to change the world by inventing something new. You just do and watch the results.
> also because they are willing to break the rules that are set forth by this bottom 99.9%[...] they are going about this imposition of their will on the world by doing it all themselves, through their own hard work.
I think all these wildly successful neo-feudalists get increasingly emboldened the more they get away with bigger and bigger social infractions.
It's also clear that they haven't experienced existed an environment with extreme inequality - it's not safe for anyone there! They think the NPC plebs will continue to follow "the rules" ad perpetuam without considering that it is a direct result of the stability they are actively undermining. They clearly don't read enough history.
What makes it “Libertarianism” still? To me it feels like they’re taking away freedom, control, and influence from everyone who is not them. Even the concentration of wealth is itself taking away everyone else’s places in the world.
> What makes it “Libertarianism” still?
Its libertarian because it is fundamentally about individuals acting on their own without going through the government, ect. It is an individualistic framework. Individuals going about achieving their goals, even through powerful corporations, falls squarely within what libertarians support.
Yes, you can make some philosophical point about how if corporations are powerful enough, how is that in any way different from governments.
But, powerful corporations controlling society, in some sort of fallout style or bioshock style dystopia clearly describes a libertarian dystopia, not a left leaning or even fascist dystopia.
Scratch a libertarian and a fascist bleeds libertarianism here, no?
He had a video on Youtube where he proudly gloated about how he voted for Trump in not one but two elections, how happy he is that he can now openly talk about it, how its a fresh start for US, how catastrophic Harris would have been.
Did he take down the video because of embarrassment or did he fear negative impact on his sales?
Damn, that's a take.
Geohot has always been an arrogant cunt who thinks he's better than everyone else. That blog post is totally on brand.
For those unaware, Mencius Moldbug is the pen name of Curtis Yarvin, thought leader for the Silicon Valley branch of right-wing technofascist weirdos which includes Peter Thiel and apparently half of a16z.
I just backed their TINY on Kickstarter.
That thing is NOT related to tinybox or tinygrad in any way. It is basically copyright infringement. Unless you’re astroturfing here I suggest you get your money back.
Wasn’t astroturfing, I’ll look into it, thanks.
Sorry for even mentioning astroturfing, haha. It’s just because the promotion of the device is based on trying to fool people it was made by tiny corp.
In my case they apparently succeeded.
Give me token/s for favourite models.
I thought there was a typo in the price
I wonder how much has he sold.
Surprising to see this with AMD GPUs considering how George famously threw up his hands as AMD not being worth working with.
Yeah, and labeling AMD "Driver Quality" as "Good" (for comparison, they label nvidia's driver quality as "Great").
Things changed. On my new Ryzen Strix Halo laptop I was able to run training experiments with PyTorch on Windows day 1: https://news.ycombinator.com/item?id=46052535
Yeah, installing random wheels from non official sources is an improvement. Not sure I’d characterize that as an unmitigated win. But also as soon as you try to do more involved things, at least personally, I ran into serious challenges getting things to work.
How does this thing cool down?
Finally, a computer that should be able to run Monster Hunter Wilds with decent performance.
But let’s be real, 12k is kinda pushing it - what kind of people are gonna spend $65k or even $10M (lmao WTAF) on a boutique thing like this. I dont think these kinds of things go in datacenters (happy to be corrected) and they are way too expensive (and probably way too HOT) to just go in a home or even an office “closet”.
It’s not for people to buy. It’s for companies to buy. Compare to salary, and it’s cheap.
> What's the goal of the tiny corp? To accelerate. We will commoditize the petaflop and enable AI for everyone.
I had the same feeling as throwadem when reading this. Your comment clarify what they meant by "everyone"
Hm, I compared my salary with $10M and it doesn't feel cheap. I guess skill issue.
But how will I make ad-supported youtube videos about how I automated my life with OpenClaw using a $10M boutique AI server to make a few thousand in ad revenue while burning tens of thousands per month on API cost.
What companies are buying this instead of like a Dell server or whatever?
These specs look enormously cheaper than doing it with dell servers. The last quote I had for a bog standard dell server was $50k and only if bought in the next few days or so. The prices are going up weekly.
So what’s the catch? If it seems too good to be true it probably is.
These are "unsupported" configurations. Nvidia/AMD discourage running multiple gaming/workstation cards and encourage customers to buy $500K SXM/OAM servers.
Who is this for?
"tiny" and it's 20k lbs and cost about 10k...
Since when did our perception of tiny blow out of size in tech? Is it the influence of "hello world" eletron apps consuming 100mb of mem while idle setting the new standard? Anyway being an AI bro seems like an expensive hobby...
> Can I pay with something besides wire transfer? In order to keep prices low and quality high, we don't offer any customization to the box or ordering process. Wire transfer is the only accepted form of payment.
Sorry, what? Is this just a scam?
Wire transfer has no comission or extra costs associated to it, so I find it very honest.
man, cmon. a little more effort.
Sure thing. For those who don’t know, wiring money like this is a good way to lose your money.
https://consumer.ftc.gov/articles/what-know-you-wire-money
Wire transfer is a bank transfer, not money wire to Western Union and like.
Yeah I agree the FTC article could be more clear here. I think they call out Western Union because those are tools that are commonly used by scammers.
But let’s be clear: the risks are the same if you are wiring money through Western Union or wiring through any other bank. Once you wire the money you do not have the same protections as other payment mechanisms. And if you don’t get the product as described, you are likely out your money. This is compared to other forms of payment like credit cards where you are protected. With a credit card you can issue a charge back to the seller and get your money back in the case of fraud. With a wire transfer you cannot.
Is this real? Reads like a joke. They sell a $12K machine, a $60K machine, and a $10M machine???
Nvidia has $4K DGX Spark, $120K DGX Station, $500K DGX, and $7M NVL72.
[dead]
[dead]
[dead]
[dead]
[flagged]
[flagged]
[flagged]
[dead]
[flagged]
Old news. ROCm works a lot better now than it did a year ago.
You are still really limited in what you can run. So much stuff is cuda only.
Like what? Most of the good stuff is ported over already and anything else, tag Anush on X and see what you get. Also happy to help.
The point is that they care now.
Tbh my experience is in the non AI uses, recently I was looking at Gaussian splatting tools and it seemed the majority of it was CUDA only. I’m also still bothered AMD for ages claimed my card (5700xt) would be getting rocm but just abandoned it.
>I was looking at Gaussian splatting tools and it seemed the majority of it was CUDA only.
Not surprising. True, the ecosystem is like early OSX vs. Windows. Eventually it'll get ported over if there is demand.
trl. give me a uv command to get that working.
But even in the amd stack things (like ck and aiter) consumer cards are not even second class citizens. They are a distance third at best. If you just want to run vllm with the latest model, if you can get it running at all there are going to be paper cuts all along the way and even then the performance won't be close to what you could be getting out of the hardware.
It is not perfect, but it isn't that bad anymore. Tons of improvements over the last year.
[dead]
[dead]
[dead]
[flagged]
"but if you haven't contributed to tinygrad your application won't be considered" this company expects people to work for free?
> See our bounty page to judge if you might be a good fit. Bounties pay you while judging that fit.
Literally the line above that
They MIGHT pay you IF you're a fit. They're bounties, i.e. spec work. They also pay a max of $1000, most of them significantly less. You can see more info at the link in that line:
> All bounties paid out at my (geohot) discretion. Code must be clean and maintainable without serious hacks.
No thanks. If you want to try before you buy, have your candidates do a paid test project. Founders need to stop acting like it's a privilege to work for them. Any talent worth hiring has plenty of other options that will treat them with respect.