# Hy3 GGUF

URL: https://interfaze.ai/models/angelslimhy3-gguf

Hy3 GGUF by AngelSlim, a text-generation model. Understand and compare features, benchmarks, and capabilities.

## Comparison

| Feature | Hy3 GGUF | Interfaze |
| --- | --- | --- |
| Input Modalities | text | image, text, audio, video, document |
| Native OCR | No | Yes |
| Long Document Processing | Yes | Yes |
| Language Support | unknown | 162+ |
| Native Speech-to-Text | No | Yes |
| Native Object Detection | No | Yes |
| Guardrail Controls | No | Yes |
| Context Input Size | 1M | 1M |
| Tool Calling | Yes | Tool calling supported + built in browser, code execution and web search |

### Scaling

| Feature | Hy3 GGUF | Interfaze |
| --- | --- | --- |
| Scaling | Self-hosted/Provider-hosted with quantization | Unlimited |

[Try Interfaze](https://interfaze.ai/dashboard)[Read the Docs](https://interfaze.ai/docs)

View model card on [Hugging Face](https://huggingface.co/AngelSlim/Hy3-GGUF)

Quantize and run **Hy3 (hy\_v3)** on llama.cpp with MTP self-speculative decoding and a thinking/tool-call parser. One script patches and builds llama.cpp; recipes and a chat template are included for quantizing your own low-bit GGUF from a calibration set.

Two parts below: **Deploy** (build & run) and **Quantization**.

* * *

## Build

```
bash setup_hyv3_llama.sh                    # clone into ./llama.cpp-hyv3, auto CUDA/CPU
bash setup_hyv3_llama.sh /path/to/target    # choose the clone directory
CUDA=0 bash setup_hyv3_llama.sh             # force a CPU-only build
```

The script pins a verified llama.cpp base commit, applies `patches/01` (base arch) then `patches/02` (MTP + parser), and builds it. Binaries land in `<target>/build/bin/`.

## Run

```
./llama.cpp-hyv3/build/bin/llama-server -m /path/to/Hy3.gguf -ctk q8_0 -ctv q8_0 -fa on -c 65536


./llama.cpp-hyv3/build/bin/llama-server -m /path/to/Hy3-mtp.gguf --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-n-min 1 -ctk q8_0 -ctv q8_0 -ctkd q8_0 -ctvd q8_0 -fa on -c 65536
```

### Recommended setups

| GPUs | build | MTP | \-c (context) | KV cache |
| --- | --- | --- | --- | --- |
| 1× H20 (96 GB) | IQ1\_M | no | \-c 65536 | \-ctk q8\_0 -ctv q8\_0 |
| 2× H20 (192 GB) | IQ1\_M | yes | / (default) | / (f16) |
| 2× H20 (192 GB) | Q4\_K\_M | yes | \-c 65536 | \-ctk q8\_0 -ctv q8\_0 -ctkd q8\_0 -ctvd q8\_0 |
| 4× H20 (384 GB) | Q4\_K\_M | yes | / (default) | / (f16) |

Weights: IQ1\_M ~83 GiB, Q4\_K\_M ~166 GiB (MTP adds ~2 GiB plus a draft KV cache). 1 card fits only IQ1\_M, and tightly — shrink context, quantize KV, no MTP. 2 cards run IQ1\_M+MTP with room to spare (drop `-c`/KV flags), but Q4\_K\_M+MTP is tight, so keep `-c 65536` and q8\_0 KV (main + draft). 4 cards run Q4\_K\_M+MTP comfortably.

## Troubleshooting

-   **`Error: no such instruction: vdpbf16ps`** (in `ggml-cpu`'s `vec.cpp`/`sgemm.cpp`): the CPU has AVX512-BF16 but the system assembler (old binutils) can't encode it. The script auto-detects this and adds `-DGGML_NATIVE=OFF`; force it with `GGML_NATIVE=0 bash setup_hyv3_llama.sh` if needed. Unrelated to the patches; CUDA inference is unaffected.
-   **`undefined reference to SSL_get1_peer_certificate`**: the system OpenSSL is too old. OpenSSL is off by default here (`-DLLAMA_OPENSSL=OFF`); it only affects the server's HTTPS model download, not local GGUF serving. Set `OPENSSL=1` if you have OpenSSL 3.0+ and need it.

* * *

If you have a calibration set, you can compute your own importance matrix and quantize the model yourself. All steps use stock llama.cpp tools; the mixed-precision recipes and a minja-compatible chat template are shipped in this folder.

### The mixed-precision recipes

The recipes in `recipes/` spend bits where they matter:

-   **Attention** (`attn_q/k/v`) and the **token embedding / output head** stay high (q8\_0 / q4\_K / q6\_K) — cheap in size, and where low bits hurt most.
-   **Shared experts** (`ffn_*_shexp`, active on every token) stay at q5\_K–q6\_K.
-   **Routed experts** (`ffn_*_exps`) carry the aggressive low bits and dominate the file size. In the IQ1\_M recipe most layers are iq1\_m, with `ffn_down` a bit higher (iq3\_xxs) and sensitive layers upgraded (iq2\_xxs) layer-by-layer.

| recipe | target | MTP head |
| --- | --- | --- |
| recipes/hyv3\_q4km\_recipe.txt | ~Q4\_K\_M mixed | no |
| recipes/hyv3\_q4km\_mtp\_recipe.txt | ~Q4\_K\_M mixed | yes |
| recipes/hyv3\_iq1m\_recipe.txt | ~IQ1\_M mixed (extreme) | no |
| recipes/hyv3\_iq1m\_mtp\_recipe.txt | ~IQ1\_M mixed (extreme) | yes |

The `*_mtp` variants add the MTP block (`blk.<n>.nextn.*` and layer-`<n>` experts). Use them only for a gguf that still has the MTP head; use the plain ones for a gguf converted with `--no-mtp`. (The MTP block's experts stay at K-quant, not IQ\*, because very-low-bit IQ types need an imatrix and the imatrix only covers the trunk layers.)

### 1\. Convert HF → BF16 GGUF

```
PYTHONPATH=./llama.cpp-hyv3/gguf-py python ./llama.cpp-hyv3/convert_hf_to_gguf.py /path/to/HYV3-hf --outfile Hy3-BF16.gguf --outtype bf16
```

### 2\. Compute the importance matrix (imatrix)

The calibration file is plain text containing samples with the model's special tokens already applied (one file, fed via `-f`). The imatrix is independent of the target quant type — compute it once on BF16 and reuse it for any recipe.

```
./llama.cpp-hyv3/build/bin/llama-imatrix -m Hy3-BF16.gguf -f calib.txt -o imatrix.gguf --output-format gguf --parse-special
```

### 3\. Quantize with a recipe

Pick the recipe matching your target and whether the gguf has an MTP head (table above). `--token-embedding-type` differs: keep it at q8\_0 for Q4\_K\_M (embedding is cheap, no reason to crush it), drop it to q4\_K for the extreme IQ1\_M build. The examples below are for MTP ggufs — use the `non-mtp` recipe for a `--no-mtp` gguf.

```
./llama.cpp-hyv3/build/bin/llama-quantize --imatrix imatrix.gguf --tensor-type-file recipes/hyv3_q4km_mtp_recipe.txt --token-embedding-type q8_0 --output-tensor-type q6_K Hy3-BF16-mtp.gguf Hy3-Q4_K_M-mtp.gguf Q4_K_M


./llama.cpp-hyv3/build/bin/llama-quantize --imatrix imatrix.gguf --tensor-type-file recipes/hyv3_iq1m_mtp_recipe.txt --token-embedding-type q4_K --output-tensor-type q6_K Hy3-BF16-mtp.gguf Hy3-IQ1_M-mtp.gguf IQ1_M
```

### 4\. (Optional) Embed a minja-compatible chat template

Hy3's built-in `chat_template` uses Python `str.format`, which llama.cpp's minja engine can't evaluate → chat/`--jinja` mode crashes. `hyv3_opensource_chat_template.jinja` is a static version (suffix hard-coded to `:opensource`, `.format` expanded). Bake it into the gguf so `--jinja` works without `--chat-template-file` at runtime:

```
PYTHONPATH=./llama.cpp-hyv3/gguf-py python -m gguf.scripts.gguf_new_metadata --chat-template "$(cat hyv3_opensource_chat_template.jinja)" --force Hy3-IQ1_M-mtp.gguf Hy3-IQ1_M-mtp-tmpl.gguf
```

## Want more deterministic results?

[Try Interfaze](https://interfaze.ai/dashboard)[Read the Docs](https://interfaze.ai/docs)