Startup Ideas Bank

A technically impressive engine that ships without the parts a customer needs to actually run a service.

AI roast score: 58/100 (D)

The idea

DeepInfer: a race car engine with no wheels. No KV cache, no load balancer, no disaggregation. Built for one model, two exotic chips, and a buyer who must finish the product themselves.

In the founder's own words

deepseek-ai/DeepInfer — DeepInfer: High-performance, large-scale inference framework for DeepSeek models

DeepInfer

DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.

Supported models:

DeepSeek-V4.1-Flash

Supported features:

Optimized kernels, model and sampler runtime

Parallelism: Attention DP / balanced CP, MoE EP + EPLB

DSpark and confidence-scheduled verification

SWA bounded replay

Asynchronous scheduling (zero-overhead batch scheduler)

Batch-invariant inference

Deterministic sampling

Structured outputs


Feature details and configuration

Optimized kernels: FlashMLA , DeepGEMM , DeepEP , TileKernels , DeepSelect , and custom TileLang kernels .

DSpark and confidence-scheduled verification: accelerate decoding with draft tokens and use confidence estimates to schedule verification. Set DEEPINFER_ENABLE_MTP_VARLEN=1 ; DEEPINFER_USE_NEXTN_DECODE controls the maximum draft length (default: 5).

Decoder SWA bounded replay: reduce decoder prefill work by replaying only a bounded suffix. Set DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 for the default V4.1 settings; the retained suffix must cover the SWA window plus the maximum backtracking length.

Asynchronous scheduling: overlap CPU batch scheduling with device execution. Enabled by default.

Batch-invariant inference: keep model results identical across different batch compositions. Enable with DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only).

Deterministic sampling: produce identical outputs for identical input tokens and sampling parameters. Set DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0 and pass a fixed seq_seed to SamplingParams (NVIDIA only).

Structured outputs: constrain generation to a regex, JSON schema, or EBNF grammar. Pass so_args={"schema_type": ..., "schema": ...} to SamplingParams ; see examples .


Supported hardware:

NVIDIA SM100 GPUs

Huawei Ascend 950 NPUs

Required for a complete inference service (not included in this release):

A KV cache system

VE/Prefill/Decode disaggregation (most required engine functionality is already implemented)

Load balancing across data-parallel replicas for both prefill and decode

Getting started

Prerequisites


Linux

GCC >= 13.0

Python 3.12

CUDA 13 (NVIDIA only)

CANN 9.2 (Ascend only)

Build from source

Before building DeepInfer, please create and activate an empty Python 3.12 environment.

# Use `ascend` instead for the Ascend platform
# Keep it exported in the environment where you run DeepInfer
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

Offline inference

Th

The roast

This is a serious engineering artifact, not a business. The README itself says a complete service needs "A KV cache system", "VE/Prefill/Decode disaggregation" and "Load balancing across data-parallel replicas", and "not included in this release". So the buyer gets the hard-to-sell half. Enterprises paying $100k-$1M want something that runs on day one. The support list is narrow. One model, "DeepSeek-V4.1-Flash", and two hardware targets, "NVIDIA SM100 GPUs" and "Huawei Ascend 950 NPUs". Most mid-market buyers do not own SM100 chips yet. Ascend chips are hard to buy outside China, which clashes with your global plan. You also plan to charge premium prices and win through product-led growth, while your biggest unknown is whether you can win customers cheaply. Free open-source code with build-from-source steps (GCC 13, Python 3.12, CUDA 13) attracts hobbyists, not enterprise buyers. vLLM and SGLang are free, mature and already everywhere.

Red flags

Verdict

Finish the missing service pieces or sell a managed version, because today you are giving away an engine and hoping someone pays for the car.

Start with your own startup idea →

More ideas roasted by AI

Browse all startup ideas →