Qwen3.6 35B A3B Escha W2

By Escha Labs Inc.

Escha-W2 is a 2-bit quantized build of Qwen3.6-35B-A3B, a Mixture-of-Experts model with 256 experts, packaged with everything needed to serve it locally through an OpenAI-compatible HTTP API. (The runtime and served model id keep the escha name — see Connecting a client.)

The whole thing is 12.3 GB on disk and runs on a single 24 GB consumer GPU — or on a 16 GB card (e.g. RTX 5060 Ti), where you trade concurrency or context, not both¹.

Contents

This repo holds only the model. The runtimes live in a separate repo, EschaLabs/escha-runtime-qwen3moe, one directory per engine: sglang/ (the escha wheel + serve.sh — servers, concurrency, tools, structured output) and zml/ (a single-binary runtime — no Python, no dependencies, and the stronger single-user decode on most cards).

Which engine?

Use sglang/ unless you have a specific reason not to. It is the engine every number on this page was measured with, and the only one that supports concurrency, tool calling, structured output and a reasoning parser.

Choose zml/ when you want a single-user box with no Python at all — one binary, no venv, no CUDA toolkit, 14-second install — and you generate long answers. It is the only way to run this model without a Python environment.

Two things decide whether that trade is worth it on your box:

• The decode lead is card-dependent. On answers ≥1k tokens, measured 2026-07-27: +15–26% on an RTX 5090 or 3090, +8–11% on a 5080, +2–5% on a 4090 (against a fully tuned SGLang), and a tie on a 5060 Ti. ZML loses short replies on every card, and its first start compiles graphs — 75–145 s on a 4090, minutes on a 16 GB card, versus ~33 s for SGLang.
• The lead is greedy-only. ZML's fast path fuses 16 decode steps per GPU dispatch with on-device argmax; any temperature > 0 falls back to a per-token loop that is ~2.15× slower (RTX 4090, 512-token answer: 218 tok/s at temperature: 0 vs 102 at 0.6). Send "temperature": 0 for the quoted numbers — including with opencode.json below, which sets 0.6. The SGLang engine samples at full speed.

Quickstart

Install the Escha runtime first (wheel + serve.sh + full detail, including two requirements that fail quietly) — then download this model and serve it:

python3.12 -m venv .venv && source .venv/bin/activate
pip install -U pip wheel "huggingface_hub[cli]"




pip install "torch==2.9.*" --index-url https://download.pytorch.org/whl/cu128



hf download EschaLabs/escha-runtime-qwen3moe --include "sglang/*" --local-dir .
pip install ./sglang/escha-*.whl     # pulls transformers>=5.8 + the full dep closure


hf download EschaLabs/Qwen3.6-35B-A3B-Escha-W2 --local-dir ./escha-w2

MODEL=./escha-w2 bash sglang/serve.sh

Sanity-check the stack before serving — the first line must print three Trues:

python -c "import torch, escha, sglang; print(torch.cuda.is_available(), hasattr(torch.ops.escha, 'escham_moe_linear'), bool(sglang.__version__))"
python -c "import escha; print(escha.__version__)"     # paste this into bug reports

Then:

curl -s http://127.0.0.1:30000/v1/models | python3 -m json.tool

Connecting a client

The server speaks the standard OpenAI API, so anything that can point at a custom base URL works — LM Studio, opencode, Open WebUI, the openai Python package, plain curl.

opencode.json in this folder is a ready-made config for opencode — copy it to ~/.config/opencode/opencode.json. It sets temperature: 0.6 (Qwen's recommended sampling for answer quality); on the ZML engine that halves decode speed — set 0 there if you want the headline throughput instead.

If you expose the server beyond your own machine, set HOST=0.0.0.0 and API_KEY=..., and put it behind a VPN or tunnel rather than opening the port directly to the internet.

Thinking mode

The model can reason before answering. Toggle it per request via chat_template_kwargs — a top-level enable_thinking field is ignored:

{
  "model": "escha-qwen36-35b-a3b-w2",
  "messages": [{"role": "user", "content": "..."}],
  "chat_template_kwargs": {"enable_thinking": true}
}

With thinking on, the reasoning arrives in reasoning_content and the answer in content — read both. To turn thinking off for the whole server instead, start it with THINK=0.

usage.reasoning_tokens reports how much of the answer went to reasoning — use it to size a thinking budget.

Tuning

serve.sh documents every knob at the top of the file. The ones that matter most on a 24 GB card:

For per-architecture (incl. the RTX 50-series ATTN_BACKEND=triton knob) and per-VRAM (16 / 24 / 40 GB+) launch recipes, see the runtime's "Running on your GPU" cookbook.

Verified configurations

Each command below was run on 2026-07-27 on the first card named in its heading, and produced the numbers in Performance across GPUs; a second name is a same-arch, same-VRAM sibling the recipe should carry to, not a separately measured card. Copy the line for your card. MODEL= and VENV= are omitted for brevity — set MODEL to your download directory.

RTX 4090 / L40S — 24 GB, sm_89

RADIX=0 MAXMAMBA=32 MAXREQ=32 CUDA_GRAPH_BS="1 2 4 8 16 32" \
  MEM=0.78 CTXLEN=32768 bash sglang/serve.sh          # 225 tok/s bs1 · 1,321 tok/s @ bs32

RTX 3090 / A6000 — 24 GB, sm_86 (Ampere)

RADIX=0 MAXMAMBA=16 MAXREQ=16 CUDA_GRAPH_BS="1 2 4 8 16" \
  MEM=0.78 CTXLEN=32768 bash sglang/serve.sh          # 154 tok/s bs1 · 390 tok/s @ bs16

RTX 5090 — 32 GB, sm_120

INT8=on ATTN_BACKEND=triton MEM=0.82 CTXLEN=32768 bash sglang/serve.sh    # 283 tok/s bs1

ATTN_BACKEND=triton MEM=0.82 CTXLEN=32768 MAXREQ=32 MAXMAMBA=48 RADIX=0 \
  CUDA_GRAPH_BS="1 2 4 8 16 32" bash sglang/serve.sh                      # ~2,670 tok/s @ bs32

RTX 5080 / 5060 Ti — 16 GB, sm_120

ATTN_BACKEND=triton MEM=0.92 CTXLEN=8192 CHUNK=2048 MAXREQ=16 MAXMAMBA=16 \
  RADIX=0 GRAPHS=1 CUDA_GRAPH_BS="1 2 4 8 16" bash sglang/serve.sh



ATTN_BACKEND=triton MEM=0.92 CTXLEN=32768 CHUNK=2048 MAXREQ=2 MAXMAMBA=2 \
  RADIX=0 GRAPHS=1 CUDA_GRAPH_BS="1 2" bash sglang/serve.sh

8k is the recipe's per-request cap, not the card's ceiling¹ — context and concurrency draw on one shared KV pool, so a 16 GB card gives up one or the other, and which one is your call.

Three settings that are worth getting right

• CUDA_GRAPH_BS must list your maximum batch size. If MAXREQ=16 but the capture list stops at 8, batch 16 silently runs eager — measured −8.8% aggregate on a 5060 Ti, and it makes batch 16 look like a throughput ceiling when it is not. Costs ~2.5 s of startup and no VRAM.
• MEM=0.92, not higher, on a 16 GB card. At 0.94 a 16 GB card has ~0.43 GB of headroom and a 1024/1024 batch-16 request can OOM inside the fused MoE op and take the whole server down. 0.92 survives it and costs nothing in throughput.
• RADIX=0 unless you specifically need prefix caching. On this hybrid model the radix cache is incompatible with the overlap scheduler, so RADIX=1 silently disables it — worth ~20% single-stream decode (measured 187 → 225 tok/s on a 4090) and it also clamps concurrency via the mamba state pool. Keep RADIX=1 only for multi-turn agent workloads that genuinely reuse long prefixes.

This checkpoint is text-only (the qwen3_5_moe config declares a vision tower, but the quantized weights contain none — vision is in the quant ignore list); serve.sh sets SGLANG_VLM_TEXT_ONLY=1 by default so the tower is never instantiated. Do not send image inputs.

Requirements in detail

• GPU — 16 GB VRAM minimum (fewer streams or less context, your choice¹; e.g. RTX 5060 Ti), 24 GB recommended; NVIDIA Ampere (sm_80) through Blackwell (sm_120). The runtime ships a fat binary with native kernels for Ampere (sm_80/86), Ada (sm_89), Hopper (sm_90), and Blackwell (sm_100/sm_120), plus PTX for forward-compat on newer GPUs. The kernel launch route is auto-selected per GPU at runtime. Cards older than sm_80 (e.g. Turing) are not supported.
• An NVIDIA driver — but no CUDA toolkit. ptxas ships inside triton, which PyTorch already installs, and serve.sh probes that copy first; you do not need to install CUDA to serve this model. If discovery ever fails, point TRITON_PTXAS_PATH at a ptxas binary (not its directory — a directory is silently rejected).
• Linux x86-64 with glibc ≥ 2.28 — the wheel is tagged manylinux_2_28, so Ubuntu 20.04+ and RHEL/Rocky/Alma 8+ are fine. WSL2 is untested.
• vm.overcommit_memory=1 — on a fresh machine the first Triton compile forks a large process and fails under the default setting. serve.sh prints the one-line fix if it detects this.
• transformers >= 5.8 — ignore this file's transformers_version for pinning. The runtime wheel installs a compatible transformers for you. Do not pin transformers to "match" any version field you find in the repo's config.json — versions below 5.8 lack this architecture's config class and serve fluent-looking nonsense with only a single warning.

Format notes

• escha_s_in / escha_s_out ship as all-ones: the trained scales are already folded into escha_rin / escha_rout at export. Do not edit or "re-apply" them — non-ones values would be applied on top of the folded scales.
• escha_config (int32[9]) and quantization_config.layer_meta are informational (written by the exporter for offline tools). The runtime derives each projection's code rate from the code tensor's shape; layer_meta records it as K (2 for gate_up_proj, 3 for down_proj) with bits mirroring K.
• The mtp.* tensors (next-token-prediction head) are present in the checkpoint but not served; speculative decoding additionally needs a separate draft export that is not part of this release.

Benchmarks

Two questions matter for a 2-bit build: how much quality did it cost, and how does it serve. Quality is measured against an FP8 baseline²; serving is measured on a single RTX 4090.

Quality vs FP8

Commonsense-6³ — thinking-off, same box, paired:

Capabilities — one representative benchmark per axis, this build vs the FP8 baseline. Retention is this-build ÷ FP8.

Across the six axes this 2-bit build stays within noise of the ~3× larger FP8 baseline everywhere except long-horizon code generation (LiveCodeBench), the one capacity-bound gap. Coding parity is corroborated by two better-powered paired-McNemar ties with FP8: HumanEval+ 92.07 vs 93.9 and CRUXEval-O 61.75 vs 63.0.¹⁰

Evaluation protocol (reproduce these numbers)

Thinking mode and the token budget change reasoning scores materially, so both are stated per benchmark. Both arms (this build and FP8) always ran the identical protocol, so every Δ above is apples-to-apples.

Sampling for the thinking-on runs is Qwen's recommended thinking config: temperature=1.0, top_p=0.95, top_k=20, min_p=0, presence_penalty=1.5, n=1.

Performance across GPUs

Measured end-to-end on five consumer cards with the SGLang engine (sglang/; launch recipes per card in the runtime cookbook). Decode = what one user sees streaming; peak throughput = total server output at the best batch size.

Every row was measured on that physical card with the launch command in Verified configurations above — 2026-07-27, five independent fresh-machine evaluations, one harness.

For scale: fast reading is ~5 words/s (≈7 tok/s) — even the slowest card here decodes ~18× faster than you can read, and a 5090 serves a 32-user pool at interactive speed. Three levers matter on every card: CUDA graphs (GRAPHS=1, the default — this hybrid MoE is launch-bound eager, graphs are worth 4.4×), RADIX=0 for throughput/latency work (worth ~20% single-stream — see Verified configurations), and the INT8 knob (single-user vs batched).

Two things the spread above is not: it is not a VRAM ranking (the 16 GB 5080 beats the 24 GB 3090 by 38%), and it is not a generation ranking. It tracks memory bandwidth and SM count — this is a 2-bit model, so decode is bandwidth-bound at batch 1 and compute-bound at batch. Ampere (3090) is the slowest per stream and also the only card here where the kernel route auto-selects lovelace rather than blackwell.

Serving — NVIDIA ISL/OSL grid, five GPUs

Single-stream decode tok/s by input/output shape, on the standard NVIDIA grid¹². Five independent fresh-machine evaluations, one card each, 2026-07-27. A dash means that evaluator did not run that shape — the five chose overlapping but not identical grids, so the table is a union rather than a full matrix¹³.

Single-stream decode barely moves with prompt length. From a 128-token prompt to a 2,048-token one it falls 11% on the 5090, 2% on the 4090, 7% on the 5080, 5% on the 3090; the 4090 still holds 90% of its short-prompt rate at a 20,000-token prompt. Decode is bandwidth-bound and the KV read is small next to the weights, so what you pay for a long prompt is TTFT, not tokens per second.

Under concurrency each card's peak, with the shape and recipe that produced it:

The 4090 grid was also reproduced from scratch on a second 4090 following only these public docs: every bs-1 shape matched within 0.1–1.8%, including the 20,000-token cell to 0.1%.

Latency SLA — MLPerf-style Poisson arrivals, RTX 4090

Poisson arrivals¹⁴ — 1000-in / 400-out, 100 requests per point, nearest-rank p99:

Saturation is ~1.9 req/s (772 tok/s ÷ 400 output tokens). Past that, waits grow by ordinary queueing, not by any defect. Decode is never the limiter — TPOT keeps 2.4–45× headroom at every rate tested; the binding constraint is always TTFT/prefill¹⁵.

Licenses and attribution

This repository contains model weights only, released under the Apache License, Version 2.0 (see LICENSE).

• Model weights — a 2-bit quantized derivative of Qwen3.6-35B-A3B (Apache-2.0, Qwen); base license text at THIRD_PARTY_LICENSES/Qwen-LICENSE.txt.
• Tokenizer — Qwen (Qwen2Tokenizer), Apache-2.0.
• Chat template — from froggeric/Qwen-Fixed-Chat-Templates (Apache-2.0, inherited from Qwen), which fixes tool-calling and thinking-mode handling for this model family.

The runtime (the escha wheel: the SGLang serving fork + CUDA kernels, and their third-party licenses — SGLang, exllamav3, AQLM, and the BSD-3/MIT upstreams vendored in SGLang) is distributed separately and carries its own LICENSE + THIRD_PARTY_LICENSES/: EschaLabs/escha-runtime-qwen3moe. All bundled code there is permissive (Apache-2.0 / MIT / BSD-3-Clause) — no copyleft.