ThinkingCap Qwen3.6 27B GGUF

GGUF / llama.cpp quantizations of bottlecapai/ThinkingCap-Qwen3.6-27B — capability of Qwen3.6-27B with 50% less thinking tokens on average, achieved by finetuning Qwen3.6-27B (Qwen Team, 2026) with online reinforcement learning while preserving the original answer quality and style.

➡️ Full model description, evaluation results (multi-seed, statistically tested), recommended sampling params, and citation: see the main model card at bottlecapai/ThinkingCap-Qwen3.6-27B.

About GGUF and quantization

GGUF is a single-file model format for running LLMs locally with llama.cpp and compatible runtimes (Ollama, LM Studio, …). The quantized variants below store weights at reduced precision — e.g. ≈4.7 bits per weight for Q4_K_M instead of the 16-bit f16 source — cutting download size and memory severalfold at a small, measured quality cost.

Files

f16 is the unquantized source; Q8_0 is near-lossless; Q4_K_M is the recommended size/quality balance for most local setups.

Usage (llama.cpp)

llama-cli -hf bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF:Q4_K_M -p "Hi"


huggingface-cli download bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf --local-dir .
llama-cli -m ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf -p "Hi"

Speculative decoding (MTP)

llama.cpp can run MTP (multi-token-prediction) self-speculative decoding on these GGUFs for a decode speed-up — no separate draft model needed. Add --spec-type draft-mtp when serving:

llama-server -hf bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF:Q4_K_M --spec-type draft-mtp

Set the draft length with --spec-draft-n-max (e.g. 4). Requires a recent llama.cpp build with MTP support.

Vision (image input)

ThinkingCap is a vision-language model. Image input needs the multimodal projector mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf (in this repo) loaded alongside a text GGUF — the single f16 mmproj pairs with any of the quants above.

• LM Studio / Jan / Ollama, …: download the mmproj-*.gguf from this repo; LM Studio auto-detects it and enables the image (🖼️) button.
• llama.cpp CLI:

huggingface-cli download bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF \
  ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf --local-dir .
llama-mtmd-cli -m ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf \
  --mmproj mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf --image photo.jpg -p "Describe this image."

• llama-server: add --mmproj mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf to expose an OpenAI-compatible vision endpoint.

Expected performance

From our internal serving-validation harness (llama.cpp, single-stream, temperature 0) on a fast N=100/dataset subset of MMLU-Pro (reasoning) and RealWorldQA (vision) — a quick quant-parity + decode-speed check, not the headline accuracy evals (for the multi-seed, statistically-tested results see the main model card).

Our three quants (f16/Q8_0/Q4_K_M) stay within subset noise of f16 on accuracy, and MTP self-speculative decoding (--spec-type draft-mtp, n=4) accepts ≈3.75 tokens per verify step — a ≈1.4–1.7× per-token decode speed-up on top of the finetune's ≈50% token savings. Q4_K_M + MTP (bold) is the recommended local config. For reference we also list unsloth's Dynamic GGUFs of the base model (UD-*): same llama.cpp path, but base-model quants — so they match base accuracy and reason ≈2× longer (none of the finetune's token savings).

median tokens = median completion length; task s = median tokens ÷ single-stream tok/s (real per-request time); speedup is vs the unquantized base model in standard decoding.

MMLU-Pro (reasoning)

RealWorldQA (vision)