Qwythos 9B V2

The next iteration of Qwythos: all the reasoning of Qwythos-9B, with the looping behavior fixed. v2 keeps the deep chain-of-thought, the uncensored research posture, and the 1M-token context of its predecessor, and cleans up the rough edges that showed up in real use.

• 🔁 Looping behavior eliminated — repetition/degeneration under greedy or low-temperature decoding dropped from 6.7% → 0%. You can serve it without leaning on repetition_penalty as a band-aid.
• 🧠 Reasoning fully preserved — MMLU, GSM8K, GPQA, ARC and HumanEval are all held at (or above) the v1 level. This is a hygiene upgrade, not a capability regression.
• 🧩 MTP head restored — the native multi-token-prediction module (dropped in the previous export) is back, so config and weights agree and speculative-decoding setups work.
• 🪪 Cleaner identity — the model no longer prefaces unrelated answers with its identity; it introduces itself only when you actually ask.
• 🔓 Still intentionally uncensored for research, cybersecurity, red-teaming, biology, chemistry, pharmacology and clinical work.
• 📜 Still 1M-token context (YaRN) and the native multimodal-capable Qwen3.5 stack.

What got fixed & improved (vs. the base Qwythos)

The fix uses FTPO (Final-Token Preference Optimization): we identify the exact token that starts a repetition loop and gently train the model to prefer coherent alternatives at that one position, leaving the rest of the distribution — and therefore the model's knowledge and reasoning — untouched.

Evaluations

Measured with our internal harness (generative chain-of-thought, greedy/pass@1 unless noted; MMLU/ARC/GSM8K n=500, GPQA-diamond n=198, HumanEval n=164). Judge for the quality metric: an independent LLM grader.

Capability is preserved across the board vs. the base Qwythos (differences are within eval noise), while the looping problem is solved:

Sample generations

(temperature 0.6, top_p 0.95, top_k 20, repetition_penalty 1.05 — reasoning shown condensed)

Reasoning — the snail in the well

Code — longest palindromic substring

Technical / clinical — no over-refusal

Recommended sampling

temperature=0.6, top_p=0.95, top_k=20, repetition_penalty=1.05, max_new_tokens=16384

Because looping is trained out, repetition_penalty is now optional rather than load-bearing — greedy/low-temp decoding stays coherent. Give the model room to reason (max_new_tokens) for math/code/analysis.

Long context

Ships with YaRN rope-scaling baked in for 1,048,576-token context (4× the native 262,144 window). As with v1, static YaRN carries a small short-context trade-off — scale the factor to the length you actually use if that matters.

Model details

Training procedure

• Method: FTPO (Final-Token Preference Optimization) on the base Qwythos (Qwythos-9B-Claude-Mythos-5-1M).
• Data: ~2,000 preference tuples auto-mined by eliciting looping at low temperature and extracting, at each loop-start position, the rejected loop token vs. the model's own coherent top-k alternatives.
• Hyperparameters: LoRA r=256, α=128, lr=1.5e-5, 1 epoch, early-stopped on chosen_win ≥ 0.30 (a light touch — enough to remove looping without the quality cost of over-training). All attention + MLP projections + lm_head trained.
• MTP: the native multi-token-prediction head was restored from the Qwen3.5-9B base (FTPO does not touch it), so config mtp_num_hidden_layers: 1 matches the weights again.

Usage

from transformers import AutoModelForImageTextToText, AutoTokenizer

model_id = "empero-ai/Qwythos-9B-v2"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": "Prove that there are infinitely many primes."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(text, return_tensors="pt").to(model.device)

out = model.generate(**inputs, max_new_tokens=16384, do_sample=True,
                     temperature=0.6, top_p=0.95, top_k=20, repetition_penalty=1.05)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

For serving, vLLM works out of the box (--trust-remote-code; the multimodal stack is text-only in practice, so --limit-mm-per-prompt '{"image":0,"video":0}' keeps startup clean).

Limitations

• This is a hygiene/robustness release, not a capability jump. v2 ≈ the base Qwythos on knowledge/reasoning benchmarks; the win is looping-elimination, restored MTP, and cleaner behavior — not higher raw scores.
• HumanEval is a couple points below the raw Qwen3.5-9B base (77.4 vs 81.7) — a small, known cost of the reasoning/looping-fix fine-tuning.
• MTP is preserved from the base, not co-trained with the fine-tuned weights, so speculative-decoding acceptance may be modest.
• Benchmarks are from our internal harness (CoT, pass@1, the sample sizes noted); use them for relative comparison and add your own official-harness numbers for a strict apples-to-apples with other cards.
• Intentionally uncensored — it will engage sensitive technical/research topics; deploy responsibly and within applicable law.

Acknowledgements

Built on Qwen3.5-9B (Alibaba/Qwen). Looping fixed with FTPO (Final-Token Preference Optimization). Thanks to the Empero AI team.