Engineering · language · single GPU

A 35-billion model on one consumer card.

A 35B model would normally need a second GPU or a trip through CPU memory. veizik runs Qwen3.6-35B-A3B resident on a single 24 GB card — the whole model on the GPU, about 21 GiB, no offload — and decodes at a usable rate while it is there.

It fits

The whole model, on one card.

Not sharded, not streamed from the CPU — resident, with headroom left on a 24 GB consumer card.

21.4 GiBresident, on a 24 GB RTX 3090

194 tok/sdecode, same card (balanced)

35B / ~3Btotal parameters / active per token

Qwen3.6-35B-A3B resident in 21.44 GiB on a single NVIDIA GeForce RTX 3090 (24 GB), balanced profile — the whole mixture-of-experts model on one consumer card, with room to spare. Offered-build figure.

How it fits

Large model, small work per token.

Why 35 billion parameters run where a dense 35B never would — at a reader’s level.

It is a mixture of experts. Of the 35 billion parameters, only about 3 billion take part in any one token — so the work per token is small even though the model is large.
All the experts stay on the card. veizik keeps the full set resident in about 21 GiB and routes each token to the few experts it needs. There is no second GPU and nothing is offloaded to the CPU between tokens.
The router runs natively. Choosing experts and gathering their outputs is part of the engine’s own GPU code, so the saving in compute is not spent back on moving data around.
It is deterministic. Greedy decoding gives the same tokens on the same prompt — the output hash matched across an RTX 3090, 4090 and 5090.

How fast

Measured, per card.

Card / profile	Decode	Resident	Build
RTX 3090 (24 GB) · balanced	194.3 tok/s	21.44 GiB	Offered build
RTX 3090 (24 GB) · fast	172.7 tok/s	21.51 GiB	Offered build
RTX 5090 · balanced	449.9 tok/s	21.67 GiB	Offered build
RTX 5090 · fast	544.7 tok/s	21.74 GiB	Offered build
RTX 4090 · balanced	226.7 tok/s	—	Development build

Qwen3.6-35B-A3B, single GPU, greedy decode. Decode is single-stream tokens per second; a short 21-token prompt, 128 new tokens. Offered-build figures are from the released package (cuda-qwen-1.1.1); the RTX 4090 row is a development build, shown as a record. The full board, with method, is on Benchmarks.

Serve it

One card, several users at once.

The same resident model behind a local OpenAI-compatible server — concurrency raises the card’s total throughput.

One card, several users. The 35B model loads once and answers concurrent requests — add callers and the card’s total output goes up, no second GPU.
A drop-in OpenAI server. veizik serve answers /v1/chat/completions on 127.0.0.1; point an existing OpenAI client at it and nothing leaves the machine.

Concurrency	Total throughput	Time to first token
1 request	160.9 tok/s	183 ms
4 at once	222.4 tok/s	626 ms
8 at once	230.6 tok/s	1,001 ms

One RTX 3090, one model loaded once. Total tokens per second across all in-flight requests rises as you add concurrency — the card keeps serving more without a second GPU. Offered build (cuda-qwen-1.1.1).

Run the server

serve it on one card

# a local, OpenAI-compatible server; the model loads once and stays resident
$ veizik serve qwen36_35b_a3b --port 8000

point any OpenAI client at it

# nothing leaves the machine; several clients can call it at once
$ curl -s localhost:8000/v1/chat/completions \
$ -d '{"model":"qwen36_35b_a3b","messages":[{"role":"user","content":"Hello"}]}'

Reproduce it

One command, one card.

Run it yourself and read the memory the standard way.

The model runs from one command on a single card. Load it, run a prompt, and read the resident memory while it is loaded — the figures above are what you should see.

run it on your card

# install the engine package (weights come from their publisher)
$ veizik install qwen36_35b_a3b
# a one-off prompt, greedy decode; the run prints its decode tok/s
$ veizik run qwen36_35b_a3b --prompt "Explain mixture-of-experts in one paragraph." --max-new 128

measure resident memory

# while the model is loaded, read how much of the card it holds
$ nvidia-smi --query-gpu=memory.used --format=csv,noheader -l 2
# on a 24 GB card the whole 35B model sits at about 21 GiB — it does not grow with the prompt

Decode tok/s is printed by the run; resident memory is the standard driver query. The released package is an early-release build; exact sizes and checksums are on the download page.