I did took a native c++ approach when writing a relational transformers engine (RelativeDB). My journey was pytorch -> c++ -> Triton (lang). While C++ was more performant than Triton, I couldn't afford to optimize on every gpu. I just accepted the ~15% throughput loss for my cloud service, which honestly wasn't bad for the amount of flexibility I got out of it.
But the cpp port of vllm looks great, that'd be great if you'll maintain that. I hit the same limitations with vllm.
I went through the post because of your comment but it really doesn't look like AI slop. Can you please share why you feel like its slop and not written by a human? I can also say "should have started writing your own comments" to you and its unfalsifiable. Blanket accusations with no proof is not a good move really.
I did took a native c++ approach when writing a relational transformers engine (RelativeDB). My journey was pytorch -> c++ -> Triton (lang). While C++ was more performant than Triton, I couldn't afford to optimize on every gpu. I just accepted the ~15% throughput loss for my cloud service, which honestly wasn't bad for the amount of flexibility I got out of it.
But the cpp port of vllm looks great, that'd be great if you'll maintain that. I hit the same limitations with vllm.
I had a similar success with Model2Vec static embedder and NER inference (both GGUF, compiled for WASM), ported to plain C from ONNX Runtime.
Wasm size from 30Mb to 300kb and 1.5x speedup. It's definitely worth it for performance or distribution size.
Should have started with writing your own blog posts.
Came here to say the same. Really tiring to read these slop-infested posts, where everything has the “right shape”.
I went through the post because of your comment but it really doesn't look like AI slop. Can you please share why you feel like its slop and not written by a human? I can also say "should have started writing your own comments" to you and its unfalsifiable. Blanket accusations with no proof is not a good move really.
What you get: X is the A, Y is the B.