# Qwythos 9B V2 GGUF

URL: https://interfaze.ai/models/empero-aiqwythos-9b-v2-gguf

[All models](https://interfaze.ai/models)

Qwythos 9B V2 GGUF by empero-ai, a image-text-to-text model with multimodal capabilities. Understand and compare multimodal features, benchmarks, and capabilities.

## Comparison

| Feature | Qwythos 9B V2 GGUF | Interfaze |
| --- | --- | --- |
| Input Modalities | text, image | image, text, audio, video, document |
| Native OCR | No | Yes |
| Long Document Processing | Yes | Yes |
| Language Support | unknown | 162+ |
| Native Speech-to-Text | No | Yes |
| Native Object Detection | No | Yes |
| Guardrail Controls | No | Yes |
| Context Input Size | 1M | 1M |
| Tool Calling | Yes | Tool calling supported + built in browser, code execution and web search |

### Scaling

| Feature | Qwythos 9B V2 GGUF | Interfaze |
| --- | --- | --- |
| Scaling | Self-hosted/Provider-hosted with quantization | Unlimited |

[Try Interfaze](https://interfaze.ai/dashboard)[Read the Docs](https://interfaze.ai/docs)

View model card on [Hugging Face](https://huggingface.co/empero-ai/Qwythos-9B-v2-GGUF)

GGUF quantizations of **[empero-ai/Qwythos-9B-v2](https://huggingface.co/empero-ai/Qwythos-9B-v2)** for [llama.cpp](https://github.com/ggml-org/llama.cpp), Ollama, LM Studio, jan, KoboldCpp, and other GGUF runtimes.

**Qwythos-9B-v2 is the new and improved Qwythos** — all the deep chain-of-thought reasoning of the base Qwythos, with the **looping behavior fixed**. The looping/degeneration that showed up under greedy or low-temperature decoding is trained out (**6.7% → 0%**), the native **MTP head is restored**, and the identity prompt is cleaned up — while knowledge and reasoning are held at (or above) the base Qwythos level.

The fix uses **FTPO (Final-Token Preference Optimization)**: the exact token that _starts_ a repetition loop is identified and the model is gently trained to prefer coherent alternatives at that one position, leaving the rest of the distribution — and therefore its knowledge and reasoning — untouched.

For full training details, evaluation numbers, and sample generations, see the **[base model card](https://huggingface.co/empero-ai/Qwythos-9B-v2)**.

* * *

## What's new vs. the base Qwythos

-   🔁 **Looping behavior eliminated** — repetition under greedy / low-temp decoding dropped **6.7% → 0%**. **Greedy decoding is now safe** — you no longer need `repeat-penalty` as a band-aid.
-   🧩 **MTP head restored** — the native multi-token-prediction module is back in the `-MTP-` files, so speculative-decoding (`--spec-type draft-mtp`) works.
-   🧠 **Reasoning preserved** — MMLU / GSM8K / GPQA / ARC held at or above the base Qwythos level (see the model card).
-   🪪 **Cleaner identity** — states who it is once, only when asked.
-   🔓 Still intentionally **uncensored**, still **1M-token context** (YaRN), still **multimodal-capable** (Qwen3.5 vision tower).

* * *

## Files

### Normal text weights — trunk only (32 blocks)

| File | Quant | Size | Notes |
| --- | --- | --- | --- |
| Qwythos-9B-v2-Q4\_K\_M.gguf | Q4\_K\_M | 5.34 GiB / 5.74 GB | recommended default — smallest practical, good quality |
| Qwythos-9B-v2-Q5\_K\_M.gguf | Q5\_K\_M | 6.08 GiB / 6.52 GB | balanced quality / size |
| Qwythos-9B-v2-Q6\_K.gguf | Q6\_K | 6.95 GiB / 7.46 GB | high quality |
| Qwythos-9B-v2-Q8\_0.gguf | Q8\_0 | 8.87 GiB / 9.53 GB | near-lossless |
| Qwythos-9B-v2-BF16.gguf | BF16 | 16.69 GiB / 17.92 GB | full precision (conversion base) |

If you don't know which to pick, **Q4\_K\_M is the right starting point.**

### MTP-enabled text weights (33 blocks, `nextn_predict_layers = 1`)

These embed the restored Qwen3.5-compatible MTP head. Use them with llama.cpp builds that support MTP draft speculation (`--spec-type draft-mtp`). The MTP matrices are retained at **Q8\_0** in every quantized variant.

| File | Quant | Size | Notes |
| --- | --- | --- | --- |
| Qwythos-9B-v2-MTP-Q4\_K\_M.gguf | Q4\_K\_M + MTP | 5.50 GiB / 5.90 GB | recommended MTP default |
| Qwythos-9B-v2-MTP-Q5\_K\_M.gguf | Q5\_K\_M + MTP | 6.25 GiB / 6.71 GB | balanced quality / size |
| Qwythos-9B-v2-MTP-Q6\_K.gguf | Q6\_K + MTP | 7.14 GiB / 7.67 GB | high quality |
| Qwythos-9B-v2-MTP-Q8\_0.gguf | Q8\_0 + MTP | 9.11 GiB / 9.79 GB | near-lossless |
| Qwythos-9B-v2-MTP-BF16.gguf | BF16 + MTP | 17.14 GiB / 18.41 GB | full precision (conversion base) |

### Vision projector — for image input

| File | Size | Notes |
| --- | --- | --- |
| mmproj-Qwythos-9B-v2-BF16.gguf | 0.86 GiB / 0.92 GB | CLIP-style vision encoder + projector at BF16 native precision; required for images, pairs with any text quant above |

The vision tower is inherited **unchanged from Qwen3.5-9B** — it was frozen through both the base Qwythos SFT and the v2 FTPO fine-tune, so image behavior matches base Qwen3.5-9B. This mmproj is interchangeable with any Qwen3.5-9B `mmproj-*.gguf`.

* * *

## Hybrid-precision quantization (Gated-DeltaNet / SSM tensors)

Qwythos is a **hybrid** model — a 3:1 mix of Gated-DeltaNet linear-attention (SSM) blocks and full-attention blocks. The linear-attention state tensors are disproportionately sensitive to low-bit quantization, so the K-quants here keep them at higher precision than the surrounding weights:

| Quant | ssm\_alpha | ssm\_beta | ssm\_out |
| --- | --- | --- | --- |
| Q6\_K | Q8\_0 | Q8\_0 | Q8\_0 |
| Q5\_K\_M | Q8\_0 | Q8\_0 | Q6\_K |
| Q4\_K\_M | Q8\_0 | Q8\_0 | Q6\_K |

The remaining SSM state tensors (`ssm_a`, `ssm_conv1d`, `ssm_dt`, `ssm_norm`) are kept at **F32** by the converter. This preserves the hybrid/SSM blocks for a small (~2–4%) increase in file size over a flat K-quant. `Q8_0` and `BF16` are uniform and need no overrides.

* * *

## Quick start

### llama.cpp

```
llama-cli \
  -m Qwythos-9B-v2-Q4_K_M.gguf \
  -p "Walk through the biochemistry of how organophosphate nerve agents inhibit acetylcholinesterase." \
  -n 8192 \
  --temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 \
  -c 16384
```

Because v2's looping is trained out, `--repeat-penalty` is now optional and **greedy decoding (`--temp 0`) stays coherent.**

### Ollama

```
ollama run hf.co/empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M
```

### LM Studio / jan / KoboldCpp

Drop any `.gguf` into your runtime's model directory. Qwythos uses the standard Qwen3.5 chat template; modern GGUF runtimes load it automatically from the file.

### MTP draft speculation

```
llama-server \
  -m Qwythos-9B-v2-MTP-Q4_K_M.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 6 \
  -c 16384 --port 8080
```

MTP support requires a recent llama.cpp build. If your runtime doesn't support MTP yet, use the normal files.

* * *

## Vision (image input)

Download a text quant **plus** the `mmproj-*.gguf`, then run llama.cpp's multimodal CLI/server:

```
llama-mtmd-cli \
  -m Qwythos-9B-v2-Q4_K_M.gguf \
  --mmproj mmproj-Qwythos-9B-v2-BF16.gguf \
  --image ./photo.jpg \
  -p "Describe this image in detail." \
  --temp 0.6 --top-p 0.95 --top-k 20 -c 16384
```

**Honest note:** all Qwythos training (base SFT and v2 FTPO) was **text-only** — the vision tower was never fine-tuned, so image-grounded reasoning inherits base Qwen3.5-9B behavior and has not been independently evaluated for this release.

* * *

## Sampling recommendations

Qwythos is a reasoning model — every response opens with a `<think>...</think>` block before the answer.

| Parameter | Value |
| --- | --- |
| temperature | 0.6 |
| top\_p | 0.95 |
| top\_k | 20 |
| repeat\_penalty | 1.05 (optional in v2) |
| max\_new\_tokens | 16384 |

Unlike the base Qwythos, **v2 does not loop under greedy / low-temperature decoding** — you can use `--temp 0` for deterministic runs without repetition. The 0.6-temperature settings above still match Qwen3.5's official thinking-mode recommendations for best quality.

* * *

## Long context (1M tokens)

The GGUFs ship with YaRN rope-scaling baked in for a **1,048,576-token context window** (4× the 262,144 native). Set `-c` up to `1048576`; lower it to reduce KV-cache memory for shorter prompts. A single H100/H200-class GPU comfortably handles 256k–512k; the full 1M typically needs multi-GPU or aggressive KV-cache offload.

* * *

## Conversion & verification

-   Converted and quantized with **llama.cpp** (`convert_hf_to_gguf.py`, `llama-quantize`), architecture `qwen35`, GGUF v3.
-   MTP variants: default conversion (33-block, `nextn_predict_layers = 1`, 15 MTP tensors, MTP matrices pinned Q8\_0). Normal variants: `--no-mtp` (32-block trunk-only). mmproj: `--mmproj --outtype bf16`.
-   Hybrid-precision overrides applied per the table above.
-   Structurally verified (arch / block count / `nextn` key / per-tensor types) and smoke-tested for load + coherent generation.
-   `shasum -a 256 -c SHA256SUMS` covers all 11 artifacts.

* * *

## License & acknowledgements

Apache-2.0, inherited from Qwen3.5-9B. Shared for research and experimentation, as-is.

-   Developed and released by **Empero AI**
-   Base model: **Qwen3.5-9B** (Alibaba Qwen team)
-   Looping fixed with **FTPO (Final-Token Preference Optimization)**
-   Quantization: **llama.cpp** (ggml-org)
-   HF model: [empero-ai/Qwythos-9B-v2](https://huggingface.co/empero-ai/Qwythos-9B-v2)

## Want more deterministic results?

[Try Interfaze](https://interfaze.ai/dashboard)[Read the Docs](https://interfaze.ai/docs)