nvidia/nemotron-nano-3.5-ea2

Overview

Nemotron 3.5 Lightning is a hybrid Mamba-MoE model. It ships as two checkpoints: an NVFP4 (W4A4) ModelOpt PTQ checkpoint — the default deployment target, which fits a single DGX Spark (GB10) or a single H100 — and a BF16 reference checkpoint.

Both are served with the FlashInfer Mamba backend and the align Mamba cache mode. VLLM_USE_FASTOKENS=1 is required.

Prerequisites

• Hardware: 1x DGX Spark (GB10), 1x H100, or 1x GB200
• A vLLM nightly — the Mamba cache flags below are not in a stable release yet

docker pull vllm/vllm-openai:nightly-6f91edf96d3f3272945809c04702380053bff4de

Launch commands

NVFP4 on 1x H100:

export VLLM_USE_FASTOKENS=1

vllm serve nvidia/nemotron-nano-3.5-ea2-W4A4-PTQ-20260723 \
  --trust-remote-code \
  --quantization modelopt_fp4 \
  --kv-cache-dtype fp8 \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5

NVFP4 on 1x DGX Spark (GB10) with MTP:

export VLLM_USE_FASTOKENS=1

vllm serve nvidia/nemotron-nano-3.5-ea2-W4A4-PTQ-20260723 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1,"moe_backend":"flashinfer_cutlass"}'

BF16 on 8x H100 with expert parallel and MTP:

export VLLM_USE_FASTOKENS=1

vllm serve nvidia/nemotron-nano-3.5-ea2 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --moe-backend flashinfer_cutlass \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --max-model-len 262144 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1,"moe_backend":"flashinfer_cutlass"}'

Key flags:

• --mamba-backend flashinfer — required; the default backend has no kernel for this stack
• --mamba-cache-mode align — aligned Mamba state cache, needed for prefix caching
• --mamba-ssm-cache-dtype float16 with stochastic rounding trades a little accuracy for decode speed; drop all three flags for the FP32 default
• --quantization modelopt_fp4 — only needed on Hopper, where NVFP4 is not auto-detected
• DFlash drafting needs the separate nvidia/DFlash-Nemotron-3.5-Nano-MOPD-Intermediate draft checkpoint

Known gaps

• TP8 without expert parallel is not supported on the NVFP4 checkpoint.

Benchmarking

vllm bench serve \
  --model nvidia/nemotron-nano-3.5-ea2-W4A4-PTQ-20260723 \
  --trust-remote-code \
  --dataset-name random \
  --random-input-len 32000 --random-output-len 2000 \
  --ignore-eos \
  --max-concurrency 256 \
  --num-prompts 1024