Startup Ideas Bank
Local LLMs: Enthusiast Appeal, Mainstream Nightmare
AI roast score: 55/100 (D)
The idea
jamesob/local-llm — Everything I know about running LLMs locally
jamesob's guide to running SOTA LLMs locally
Note: nothing in this README aside from the tables was written by AI.
Have $2k burning a hole in your pocket and want some local, state-of-the-art machine
intelligence?
How about $40k?
If Dario and Altman are giving you heartburn (they should be), read on to figure out
how to run this new kind of computing locally.
In this repo you'll find
the hardware I use to run SOTA locally,
why I bought what and little-known secrets for configuring it,
how I run speech-to-text (STT) locally,
ready-to-run configuration for running models I think are good within Docker containers.
Contents
Section
TL;DR
How much are you willing to spend?
$2k gets you Qwen and good STT (pretty far!); $40k gets you almost-Opus
Base system
Last-gen EPYC + eBay DDR4 for $5.6k
GPUs
4× RTX PRO 6000, 384GB VRAM (where the money went)
c-payne switch sub-BOM
Indie PCIe switching from c-payne.com so GPUs talk peer-to-peer
GPU mount
A day of carpentry
Making the switch behave
BIOS bifurcation, link speed, ASPM
Kernel / GRUB params
iommu=off or NCCL hangs
ACS disable
Keep P2P traffic inside the switch fabric
GPU power limiting
Running $46k of silicon on a 110V circuit
Result
Gen4 line rate: 27.5/50.4 GB/s, sub-µs latency
runners/
Ready-to-run serving configs: GLM-5.2-594B : vLLM docker-compose, DCP4+MTP5, ~80 t/s @ 460k ctx
runners/stt
Ready-to-run speech-to-text config with whisper-large-v3
tools/
measure-gpu-speed.sh : P2P bandwidth/latency benchmark
Resources
rtx6kpro repo, c-payne
My setup
I was lucky/dumb enough to buy 4x RTX Pro 6000s back when they were cheaper. Because
RAM is now so expensive, I opted to build a last-gen DDR4 system to host these cards,
the parts for which I got off eBay. This allowed me to keep base system cost reasonable
while still getting a lot of VRAM.
Another somewhat unusual thing I did was to use PCIe4 switches (from
c-payne.com ). This allows the GPUs to communicate to one another
"directly" at wire speeds during the allreduce step in tensor parallelism, rather than
having to send all data through the PCI root complex. The upshot of this is reduced
latency between the cards with less of a need for expensive PCIe5 hardware.
Consequently, I'm spending money on VRAM (where it counts) rather than on a PCIe5/DDR5
base system, which is terrifically expensive as of July 2026.
My particular BOM is detailed below.
How much are you willing to spend?
~$2k
A great way to go is 2x RTX 3090s for a total of 48GB VRAM total. You can then run
Qwen3.6-27B , which is an awesome model.
You can also run SOTA speech-to-te
The roast
Your idea of running state-of-the-art LLMs locally on consumer hardware is intriguing, but it's a niche market at best. You're targeting tech enthusiasts willing to fork out thousands on hardware, which drastically limits your user base. The DIY aspect, while appealing to some, creates a significant barrier to entry for the average consumer. Furthermore, your content-driven growth strategy (q17=content_seo) is too slow to capitalize on the rapidly evolving AI landscape. The lack of funding (q14=no_funding) and a solo team (q13=solo) only exacerbate the challenges in scaling and execution.
Red flags
- Niche market appeal
- High barrier to entry
- Slow growth strategy
Verdict
Without a clear path to a broader market or significant funding, this venture is unlikely to scale beyond a small group of hardcore tech enthusiasts.
Roast your own startup idea →