Got me down a rabbit hole of trying to find the ROI of doing something like this. This is what every GM at AWS thinks about. So, a back of the napkin calculation of running at FP8 and approx 35GB. You can fit that on a g6e.xlarge @ 48GB for about $900/year at reserved capacity.
Assuming you are netting at $5.5 (post stripe fees), and maybe $100/month for API/auth box for egress, you probably need about 170 paying customers to breakeven. This is assuming light/bursty usage.
Kudos! This is a a win win for both you and customer using it as it removes the undifferentiated heavy lifting.
actually the math is more like $1500 a month on a MI300x on a highly optimized rocm/sglang image and everyone's getting 100+ tok/sec and we have 100 active people and plenty of room for others.
but were losing money right now obviously.
we did come from 4x 3090s could actually make money, but we thought a couple hundred bucks a month isnt worth a 100 degree office.
People are dumping 128k cold documents in like its their job. Some dudes hit 2 billion + tokens a week.
I wanted to start using this but I can't since I can't use any email/password combination I want. By requiring an existing account from Google/GitHub/Discord you are just contributing to corporate-owned dystopia.
i know, but im scared to own our own auth, when there's money involved... too cheap to pay for a service. So we lean on the back of the giants for now.
If it helps, there's really no reason to ever log into our site after you get an API Key. I guess until you want to make another api key
So I said, OK, let me open a Discord account, just to be able to access your service. How hard can it be, right? Wrong. Got stuck in a "Wait, are you a robot?" loop, doing dozens of captchas, and in the end had to close the browser tab. Broken piece of crap.
Got me down a rabbit hole of trying to find the ROI of doing something like this. This is what every GM at AWS thinks about. So, a back of the napkin calculation of running at FP8 and approx 35GB. You can fit that on a g6e.xlarge @ 48GB for about $900/year at reserved capacity.
Assuming you are netting at $5.5 (post stripe fees), and maybe $100/month for API/auth box for egress, you probably need about 170 paying customers to breakeven. This is assuming light/bursty usage.
Kudos! This is a a win win for both you and customer using it as it removes the undifferentiated heavy lifting.
actually the math is more like $1500 a month on a MI300x on a highly optimized rocm/sglang image and everyone's getting 100+ tok/sec and we have 100 active people and plenty of room for others.
but were losing money right now obviously.
we did come from 4x 3090s could actually make money, but we thought a couple hundred bucks a month isnt worth a 100 degree office.
People are dumping 128k cold documents in like its their job. Some dudes hit 2 billion + tokens a week.
its been fun though. you should check it out
Which provider do you use to access the MI300x?
I wanted to start using this but I can't since I can't use any email/password combination I want. By requiring an existing account from Google/GitHub/Discord you are just contributing to corporate-owned dystopia.
i know, but im scared to own our own auth, when there's money involved... too cheap to pay for a service. So we lean on the back of the giants for now.
If it helps, there's really no reason to ever log into our site after you get an API Key. I guess until you want to make another api key
So I said, OK, let me open a Discord account, just to be able to access your service. How hard can it be, right? Wrong. Got stuck in a "Wait, are you a robot?" loop, doing dozens of captchas, and in the end had to close the browser tab. Broken piece of crap.
Well, are you a robot?
Ok well I need a way to securely give you an API key for you to try it out but I don't really want to post it on HN so I'm open to ideas lol
I am definitely not a robot. I have aches in several parts of my body which are giving me painful reminders it's all flesh and blood.
I'll try opening a dedicated Github account next. Fingers crossed.
lmao dont hurt yourself
What type of hardware, GPU are you utilizing? I have the same model self hosted at home
1x MI300x , rocm/sglang , highly optimized image, FP8, FP16 Cache (this fixes looping issues with qwen3.6-35b
~100tok/s average
I will try this thing. if give free month
Hi danger dingo,
Sure, sign up and ping me in discord
[flagged]
[flagged]
[dead]
[dead]