Startup Ideas Bank

Running a 744B-parameter MoE on a consumer machine is ambitious, but the market validation is MIA.

AI roast score: 65/100 (C)

The idea

A 744B-parameter model on consumer hardware is like squeezing an elephant into a Mini Cooper. Cool tech, but who’s paying for this?

In the founder's own words

JustVugg/colibri — Run GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

Tiny engine, immense model. Run GLM-5.2 (744B-parameter MoE) on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk.

$ ./coli chat
🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
✓ pronto in 32s · residente 9.9 GB
› ciao!
◆ Ciao! 😊 Come posso aiutarti oggi?

The idea

A 744B Mixture-of-Experts model activates only ~40B parameters per token — and only ~11 GB of those change from token to token (the routed experts). So:

the dense part (attention, shared experts, embeddings — ~17B params) stays resident in RAM at int4 (~9.9 GB);

the 21,504 routed experts (75 MoE layers × 256 experts + the MTP head, ~19 MB each at int4) live on disk (~370 GB) and are streamed on demand , with a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free L2.

The engine is a single C file ( c/glm.c , ~2,400 lines) plus small headers. No BLAS, no Python at runtime, no GPU required (an opt-in CUDA tier for pinned experts exists — see below).

What's implemented


Faithful GLM-5.2 ( glm_moe_dsa ) forward — validated token-exact against a transformers oracle (teacher-forcing 32/32, greedy 20/20 on a tiny-random model with the real architecture).

MLA attention (q/kv-LoRA, interleaved partial RoPE) with compressed KV-cache : 576 floats/token instead of 32,768 (57× smaller — GLM-5.2 has 64 heads and no GQA).

DeepSeek-V3-style sigmoid router (noaux_tc, routed_scaling_factor), shared expert, first-3-dense layers.

Native MTP speculative decoding — GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. The head must be int8 (the converter does this by default): at int4 draft acceptance collapses to 0–4% and speculation never engages; at int8 it's 39–59% acceptance, 2.2–2.8 tokens/forward (community-measured, #8 ). Lossless — and stays lossless under sampling via rejection sampling. Honest caveat from the same measurement: on a cold cache each verified draft routes to extra experts (~660 → ~1100 expert-loads/token), so speculation can be a net time loss until the cache/pin warms up — the adaptive guard and DRAFT=0 are there for that.

True sampling — temperature + nucleus, defaults tuned for int4 reality (0.7 / 0.90; the official 1.0 / 0.95 samples quantization noise from the tail).

Integer-dot kernels (Q8_0-style int8 activations, AVX2 maddubs ): int8 matmuls 1.4–2.5× faster (119 GFLOP/s measured), int4 1.8× in batch — routing decided per shape by measurement (int4 single-row stays f32: it measured slower).

MLA weight absorption (DeepSeek trick) for dec

The roast

Your idea to run a 744B-parameter model on a consumer machine is technically impressive but commercially unrealistic. The idea hinges on a niche technical feat that might not translate into broad consumer demand. A massive model like this is more of a novelty than a necessity for the average user, and you’re banking on people willing to pay for something they didn’t even know they needed. Also, with no funding and a solo team, the execution risk is off the charts. You’re targeting a general, global audience (q5=general, q6=global) with a pricing model that doesn’t necessarily fit the tech-savvy niche (q9=standard). Plus, without any funding (q14=no_funding) and with the biggest unknown being whether anyone will pay for this (q15=will_pay), you are essentially flying blind. Your moat is speed (q11=speed), but speed alone won’t cover the gap in market validation and consumer adoption.

Red flags

Verdict

Focus on finding a real market need before diving deeper into tech development.

Start with your own startup idea →

More ideas roasted by AI

Browse all startup ideas →