# GLM 5.3 SLIM GGUF

URL: https://interfaze.ai/models/scantistglm-53-slim-gguf

[All models](https://interfaze.ai/models)

GLM 5.3 SLIM GGUF by Scantist, a text-generation model. Understand and compare features, benchmarks, and capabilities.

## Comparison

| Feature | GLM 5.3 SLIM GGUF | Interfaze |
| --- | --- | --- |
| Input Modalities | text | image, text, audio, video, document |
| Native OCR | No | Yes |
| Long Document Processing | Yes | Yes |
| Language Support | unknown | 162+ |
| Native Speech-to-Text | No | Yes |
| Native Object Detection | No | Yes |
| Guardrail Controls | No | Yes |
| Context Input Size | 1M | 1M |
| Tool Calling | No | Tool calling supported + built in browser, code execution and web search |

### Scaling

| Feature | GLM 5.3 SLIM GGUF | Interfaze |
| --- | --- | --- |
| Scaling | Self-hosted/Provider-hosted with quantization | Unlimited |

[Try Interfaze](https://interfaze.ai/dashboard)[Read the Docs](https://interfaze.ai/docs)

View model card on [Hugging Face](https://huggingface.co/Scantist/GLM-5.3-SLIM-GGUF)

**A GLM-5.3-class model that runs on a single NVIDIA B200, or on a Mac Studio with 192 GB or more of unified memory (with mlx-serve).** At 179.9 GB it is 24.6% smaller than the 256-expert UD-IQ2\_M (238.6 GB), which does not fit in the memory of one B200 or a 192 GB Mac. On one B200 the whole model sits on the GPU and about 100K tokens of KV cache are left for up to 10 parallel requests; in our tests it scores 92.7% on HumanEval.

[GLM-5.3](https://huggingface.co/zai-org/GLM-5.3) with a **192-expert MoE architecture found by neural architecture search (NAS)**: every MoE layer keeps 192 of the original 256 routed experts. The model is built from Unsloth's [`UD-IQ2_M`](https://huggingface.co/unsloth/GLM-5.3-GGUF/tree/main/UD-IQ2_M) GGUF quantization.

|  | Unsloth GLM-5.3 UD-IQ2\_M | This model |
| --- | --- | --- |
| routed experts per MoE layer | 256 | 192 |
| parameters | 753B | 563B |
| file size | 238.58 GB (222.19 GiB) | 179.86 GB (167.51 GiB), -24.6% |
| average bits per weight |  | 2.56 |
| MTP (next-token prediction) layer | yes | removed |
| HumanEval pass@1 (greedy, thinking off) | not measured | 92.7% |
| GPQA-Diamond (T=1.0, reasoning\_effort=high, 65,536 tokens) | not measured | 84.8% |

## How it was made

-   **Architecture (NAS).** The 192-experts-per-layer architecture is the one found by the neural architecture search behind [GLM-5.3-SLIM-E192](https://huggingface.co/autotrust/GLM-5.3-SLIM). Each MoE layer (3-77) keeps the experts chosen by the search; the other 64 routed experts per layer are removed.
-   **No re-quantization.** In GGUF the expert axis is the slowest dimension of `ffn_{gate,up,down}_exps`, `ffn_gate_inp` and `exp_probs_b`, so each expert is one contiguous byte range. The kept experts are copied byte-for-byte, in their original order, as experts 0-191. Every tensor keeps the quant type Unsloth chose; all other tensors (attention, DSA indexer, shared experts, dense layers 0-2, embeddings, output) are byte-identical to the source.
-   **MTP layer removed** (`blk.78`), as in SLIM. Do not use MTP speculative decoding with this model.
-   **Metadata changes:** `glm-dsa.expert_count` 256 -> 192, `glm-dsa.block_count` 79 -> 78, `glm-dsa.nextn_predict_layers` 1 -> 0, `general.size_label` 256x22B -> 192x22B. Tokenizer and chat template are unchanged.
-   No retraining, no recalibration.

Quant types (unchanged from UD-IQ2\_M):

| tensors | size | types |
| --- | --- | --- |
| ffn\_down\_exps | 65.9 GiB | IQ3\_XXS (71 layers), IQ4\_XS (4) |
| ffn\_gate\_exps, ffn\_up\_exps | 2 x 43.7 GiB | IQ2\_XXS (74), IQ2\_S (1) |
| attn\_output, attn\_q\_a | 5.7 GiB | Q5\_K, Q6\_K |
| attn\_q\_b, attn\_k\_b, attn\_v\_b, attn\_kv\_a\_mqa, indexer | 4.7 GiB | Q8\_0 |
| shared experts | 1.9 GiB | Q5\_K / Q6\_K / Q8\_0 |
| token\_embd / output | 0.6 / 0.5 GiB | Q5\_K / Q4\_K |

## Files

| file | size |
| --- | --- |
| GLM-5.3-UD-IQ2\_M-E192-00001-of-00006.gguf | metadata only |
| GLM-5.3-UD-IQ2\_M-E192-00002-of-00006.gguf ... 00006 | 38.1 / 37.5 / 37.5 / 37.5 / 29.1 GB |
| imatrix\_unsloth\_E192.gguf | 0.86 GB. Unsloth's imatrix with the same expert slicing applied, for re-quantizing this 192-expert model to other types. Not needed to run the model. |
| eval/serve\_b200.sh | starts llama-server on one B200 with the settings used for the evaluation |
| eval/gpqa.py | GPQA-Diamond client with KV-pool admission control (see step 7 of the deployment guide) |
| eval/humaneval.py | HumanEval pass@1 client (greedy, thinking off); runs the generated code against the official tests |

## Deploying on a single NVIDIA B200

Everything below was run on one NVIDIA B200 (183,359 MiB reported by `nvidia-smi`) with llama.cpp commit `8e5a58d` (2026-10-24), NVIDIA driver 595.91 and CUDA 13.0.

### 1\. Requirements

-   One NVIDIA B200. The weights alone need about 167 GiB of GPU memory, so smaller GPUs need a multi-GPU split (not tested here).
-   An NVIDIA driver that supports CUDA 13, and the CUDA 13 toolkit (`nvcc`) to build llama.cpp.
-   180 GB of free disk space for the GGUF files.
-   For the evaluation script: Python 3.10+ and `pip install aiohttp datasets`.

### 2\. Build llama.cpp with CUDA

```
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 8e5a58d              # tested commit; newer commits may work but were not tested
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=100 -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server llama-cli -j
```

`CMAKE_CUDA_ARCHITECTURES=100` builds only for the B200 (sm\_100), which keeps the build short.

### 3\. Download

```
pip install -U huggingface_hub
hf download Scantist/GLM-5.3-SLIM-GGUF --include "GLM-5.3-UD-IQ2_M-E192-*.gguf" "eval/*" --local-dir GLM-5.3-E192
```

Always pass the first split (`...-00001-of-00006.gguf`) to llama.cpp; it loads the other splits itself.

### 4\. Memory budget

| item | GPU memory |
| --- | --- |
| weights (-ngl 99) | 170,896 MiB (token\_embd, 624 MiB, stays in host memory) |
| compute buffer (-ub 512) | ~512 MiB |
| KV cache, f16 | 93 KiB per token (MLA 87.75 KiB + DSA indexer 5.25 KiB) |
| KV cache, q4\_0 (-ctk q4\_0 -ctv q4\_0) | 26.2 KiB per token |

Measured total use after start-up, with the unified KV pool:

| \-c (KV pool, tokens) | GPU memory used | free |  |
| --- | --- | --- | --- |
| 102,400 | 181,678 MiB | ~1.7 GB | recommended |
| 110,592 | 182,454 MiB | ~0.9 GB | too little room for temporary buffers |
| 343,040 with q4\_0 KV | 182,226 MiB | ~1.1 GB | 5 requests of 68,608 tokens at once |

So a B200 holds about 100K tokens of KV cache in f16, or about 340K tokens in q4\_0, shared by all requests.

### 5\. Start the server

`eval/serve_b200.sh` starts llama-server with the settings used for the GPQA run (10 slots, 102,400-token shared pool, up to 68,608 tokens per request):

```
cd GLM-5.3-E192/eval
LLAMA_SERVER=/path/to/llama.cpp/build/bin/llama-server ./serve_b200.sh
```

This is the same as:

```
llama-server -m GLM-5.3-UD-IQ2_M-E192-00001-of-00006.gguf \
  -ngl 99 -kvu -c 102400 --kv-unified-per-slot 68608 -np 10 -bs \
  --host 127.0.0.1 --port 8080 --metrics
```

| flag | meaning |
| --- | --- |
| \-ngl 99 | put all layers on the GPU |
| \-kvu -c 102400 | one KV pool of 102,400 tokens shared by all slots |
| \--kv-unified-per-slot 68608 | limit per request: the longest GPQA prompt (2,821 tokens) + 65,536 generated tokens |
| \-np 10 | 10 parallel slots |
| \-bs | sample on the GPU (see the notes below) |

Change any value with an environment variable (`CTX`, `NP`, `SLOT_CTX`, `HOST`, `PORT`, `MODEL`); `DRY_RUN=1` prints the command without running it. Start-up takes about 30 s when the files are in the page cache; the first start also reads 180 GB from disk.

Check that the server is up, then send a request:

```
curl http://127.0.0.1:8080/health          # {"status":"ok"}

curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role": "user", "content": "Write a Python function that checks whether a string is a palindrome."}],
  "temperature": 1.0, "top_p": 0.95, "top_k": 0, "min_p": 0,
  "chat_template_kwargs": {"reasoning_effort": "high"},
  "max_tokens": 8192
}'
```

For an interactive session instead of a server:

```
llama-cli -m GLM-5.3-UD-IQ2_M-E192-00001-of-00006.gguf -ngl 99 -c 16384 --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0
```

### 6\. Choosing `-c` and `-np`

With `-kvu`, the per-slot limits may add up to more than the pool. **If the pool runs out, llama-server fails all running requests** with `Context size has been exceeded` and clears them. Size the setup by the total number of tokens that are live at the same time:

-   Short requests (chat, code): more slots are fine as long as `slots x longest request` stays below the pool, e.g. `-np 32` with `-c 102400` when no request goes past ~3K tokens. Total generation speed rises with the number of parallel sequences (see Throughput below).
-   Long reasoning (up to 65K tokens per request): one such request alone uses two thirds of the pool. Either run few slots, or let the client control admission, as `eval/gpqa.py` does: it starts a new request only when the pool has room, and when the pool is nearly full it aborts the youngest request and runs it again later.

### 7\. Run GPQA-Diamond

With the server from step 5 running:

```
cd GLM-5.3-E192/eval
python gpqa.py --url http://127.0.0.1:8080 --pool 102400 --max-conc 10 --out gpqa_results.jsonl
python gpqa.py --summary --out gpqa_results.jsonl
```

`--pool` and `--max-conc` must match the server's `-c` and `-np`. The script prints one line per finished question (accuracy so far, active requests, pool use, tokens per second, number of preemptions). Results are appended to `--out` after every question, so a stopped run continues where it stopped when you start it again.

For long-reasoning runs like this one, use 5 slots with a 4-bit KV cache instead. The pool then holds 5 requests of full length, so the client never has to abort anything:

```
CTX=343040 NP=5 ./serve_b200.sh -ctk q4_0 -ctv q4_0
python gpqa.py --url http://127.0.0.1:8080 --max-conc 5 --pool 1000000 --reserve 0 --out gpqa_results.jsonl
```

(`--pool 1000000 --reserve 0` turns the client-side pool control off; it is not needed when the server pool holds `--max-conc` full-length requests.) Our run took about 13.5 hours: the first 166 questions ran with the 10-slot f16 setup above, and once only the long, hard questions were left the client aborted and restarted requests 275 times, which wasted most of that time. The last 32 questions ran with the 5-slot q4\_0 setup in 2.3 hours, without aborts.

### 8\. Notes

-   **Reasoning.** The chat template always opens a `<think>` block. Set the effort per request with `"chat_template_kwargs": {"reasoning_effort": "low" | "high"}`; anything else means `max`. `--reasoning-budget 0` turns thinking off.
-   **Sampling.** The recommended setting is `temperature=1.0, top_p=0.95`; llama-server takes these defaults from the GGUF metadata, but it also applies `top_k=40` and `min_p=0.05` by default. Pass `top_k=0, min_p=0` to turn them off.
-   **`-bs` (backend sampling, experimental)** moves sampling to the GPU: +23% throughput at 10 slots in our test (66.9 -> 82.5 tok/s). The sampled distribution matched the CPU sampler (two-sample chi-square, p = 0.88 / 0.19 / 0.18 on three prompts), but the token probabilities returned with `n_probs` were not accurate. Leave out `-bs` if you need logprobs.
-   **No MTP.** The model has no MTP layer, so do not enable MTP speculative decoding.

### 9\. Troubleshooting

| symptom | cause and fix |
| --- | --- |
| ggml\_cuda\_init: failed to initialize CUDA: system has unsupported display driver / cuda driver combination, while nvidia-smi works | The loader picks the libcuda.so.1 of the CUDA forward-compatibility package instead of the driver's. Put the directory with the driver's libcuda.so.<driver version> first in LD\_LIBRARY\_PATH. |
| out of memory at start-up | Lower -c (each 10K tokens of pool is about 0.9 GiB). |
| Context size has been exceeded, and all running requests fail | The shared pool ran out; see step 6. |
| a reply stops with finish\_reason: length before max\_tokens | Prompt + max\_tokens is larger than --kv-unified-per-slot; raise it. |
| GGML\_ASSERT(\*cur\_backend\_id != -1) while the context is created | No GPU is in use (CPU-only build, or CUDA failed to start). At the tested commit the CPU backend cannot run this architecture; the original 256-expert UD-IQ2\_M fails the same way. |

## Throughput (one B200, `llama-batched-bench`)

Prompt 512 tokens, 128 generated tokens per sequence, unified KV, f16 or 4-bit (`-ctk q4_0 -ctv q4_0`) KV cache:

| parallel sequences | 1 | 4 | 8 | 16 | 32 | 64 |
| --- | --- | --- | --- | --- | --- | --- |
| generation, tok/s (total), f16 KV | 34.5 | 91.4 | 110.4 | 130.3 | 164.3 | 203.3 |
| generation, tok/s (total), q4\_0 KV | 40.5 | 91.1 | 109.4 | 128.9 | 162.6 | 200.2 |
| prompt processing, tok/s, f16 KV | 648 | 672 | 643 | 596 | 524 | 573 |
| prompt processing, tok/s, q4\_0 KV | 668 | 673 | 644 | 597 | 526 | 573 |

Long prompts:

| setting | f16 KV | q4\_0 KV |
| --- | --- | --- |
| 16K-token prompt, 1 sequence, generation tok/s | 37.4 | 34.9 |
| 16K-token prompt, 4 sequences, generation tok/s (total) | 81.9 | 70.6 |
| 1 sequence at 1K tokens of context, generation tok/s | 39.9 | 37.4 |
| 1 sequence at 32K tokens of context, generation tok/s | 36.5 | 32.4 |
| 1 sequence at 63K tokens of context, generation tok/s | 34.6 | 27.8 |
| prompt processing at 32K-63K tokens, tok/s | 550 | 550 |

-   With short contexts the KV type makes no difference. With long contexts the 4-bit cache is slower per sequence: -11% at 32K and -20% at 63K tokens for one sequence, -14% for four sequences at 16K.
-   In exchange, the 4-bit cache needs 26.2 KiB instead of 93 KiB per token, so five requests of 68K tokens fit on one B200 instead of one. For long-reasoning workloads this gives much more total throughput.
-   Single-sequence numbers vary by about +-10% between runs (f16, 1K context: 34.5 and 39.9 tok/s in two runs).
-   In llama-server the speed can be lower than in `llama-batched-bench`: in the GPQA run (5-slot q4\_0 setup), the last remaining request generated about 17 tok/s at ~56K tokens of context.
-   These speeds are limited by the llama.cpp kernels for these low-bit types, not by the GPU's memory bandwidth.

## Evaluation

All results below come from our own scripts against llama-server with the settings shown. They are not comparable to the numbers in the official GLM-5.3 card. **We did not run the original 256-expert UD-IQ2\_M or the full-precision model on the same harness**, so the cost of the expert reduction itself is not measured here; for that, see the A/B results of GLM-5.3-SLIM-E192 (FP8) against GLM-5.3 (FP8).

| benchmark | setting | result |
| --- | --- | --- |
| GPQA-Diamond (198 questions) | T=1.0, top\_p=0.95, top\_k/min\_p off, reasoning\_effort=high, max 65,536 new tokens, 1 sample | 84.8% (168/198) |
| HumanEval (164), pass@1 | greedy, thinking off, max 2048 new tokens, one request at a time | 92.7% (152/164) |
| HumanEval (164), pass@1 | same, 8 concurrent requests | 89.6% (147/164) |
| MMLU, 570-question subset (10 per subject, seed 0) | zero-shot, greedy, thinking off, answer letter only | 71.4% (407/570) |

-   HumanEval: answers were run against the official tests in a subprocess. The two runs differ only through batch numerics; the difference is not significant (exact McNemar test on the 9 discordant problems, p = 0.18). 11 of 164 serial answers reached the 2048-token limit (5 of them still passed).
-   GPQA-Diamond: questions from the public mirror `fingertap/GPQA-Diamond` (same 198 questions, options already shuffled). Prompt: "Answer the following multiple choice question. The last line of your response should be of the following format: 'Answer: $LETTER' ... Think step by step before answering." The answer is parsed from the final response only; a response that hits the token limit counts as wrong. The run used `eval/gpqa.py` with the server from `eval/serve_b200.sh` (step 5 and 7 of the deployment guide); requests that were aborted to keep the KV pool from overflowing were restarted from the beginning with a new seed, so every question has exactly one complete sample (48 of the 198 answers are such a later attempt). The first 166 questions ran with an f16 KV cache (10 slots, 142/166 correct); the last 32 with a q4\_0 KV cache (5 slots, 26/32 correct). 9 answers reached the 65,536-token limit and 1 ended without a parsable answer; all 10 count as wrong.

### Reproducing

HumanEval: the numbers above were produced with this server (8 slots of 8,192 tokens, thinking off, CPU sampling) and one request at a time:

```
llama-server -m GLM-5.3-UD-IQ2_M-E192-00001-of-00006.gguf -ngl 99 -c 65536 -np 8 --reasoning-budget 0 --port 8080
python eval/humaneval.py --url http://127.0.0.1:8080 --concurrency 1 --out humaneval_results.json
```

`humaneval.py` runs model-generated code on your machine; use a sandbox or a throw-away container. With other server settings the score can move by a few problems, because batch composition changes the numerics (compare the serial and the 8-concurrent run above).

GPQA-Diamond: start the server with `eval/serve_b200.sh` and run `eval/gpqa.py` (steps 5 and 7 of the deployment guide).

## Integrity checks

-   All 1,782 tensors were compared with the source UD-IQ2\_M files: non-expert tensors are byte-identical, and every expert-axis tensor equals the source tensor restricted to the kept experts, in their original order.
-   The expert mapping was cross-checked against an independent NVFP4 checkpoint built with the same expert selection (from `nvidia/GLM-5.3-NVFP4`): the kept router rows (`ffn_gate_inp`, `exp_probs_b`) are bit-identical in all 75 MoE layers, and dequantized experts match with cosine similarity 0.94 (IQ2\_XXS), 0.98 (IQ3\_XXS) and 0.994 (IQ4\_XS), against < 0.003 for a neighbouring expert.

## Limitations

-   Two losses stack: 25% of the routed experts are removed, and the weights are at ~2.6 bits per weight.
-   SLIM's A/B tests of the same expert set (FP8) show a clear loss on Chinese knowledge QA and a smaller loss on science QA; expect the same here.
-   With greedy decoding and thinking off, a few percent of long answers wander or repeat (in HumanEval, 1 of 164 answers looped on `<code>`), and the model sometimes uses `<code>` tags instead of Markdown code fences.
-   No MTP layer, so no MTP speculative decoding.

## License and credits

Released under the [GLM-5.3 License](https://huggingface.co/zai-org/GLM-5.3/blob/main/LICENSE), the same terms as the base model. Weights by [Z.ai](https://huggingface.co/zai-org); quantization and imatrix by [Unsloth](https://huggingface.co/unsloth); expert selection from GLM-5.3-SLIM-E192; runtime by [llama.cpp](https://github.com/ggml-org/llama.cpp).

## Want more deterministic results?

[Try Interfaze](https://interfaze.ai/dashboard)[Read the Docs](https://interfaze.ai/docs)