Today for AI
ALL INTELLIGENCE
Hacker News AIT2·Open Source

Unsloth Ships Dynamic v3.1 Quantizations for Qwen3.8 Model Family

Original title: Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
58AI Score

Unsloth released Dynamic v3.1 quantizations for the Qwen3.8 family, including the 27B dense model and the 2.4T-A95B MoE variant, shipping GGUF weights across UD-Q4_K_XL down to IQ1_S bit-widths. The update pairs one-line installers with ready-made llama.cpp deployment commands, lowering the barrier to running very large MoE models locally.

Related Entities

TERMDynamic QuantizationTERMGGUFTOOLllama.cppMODQwen3.8-2.4T-A95BCOUnslothMODQwen3.8-27B

Full Report

Was this helpful?

BASH
curl -fsSL https://unsloth.ai/install.sh | sh
Text
irm https://unsloth.ai/install.ps1 | iex
Text
brew install --cask unsloth
Text
brew upgrade --cask unsloth
Text
winget install --id UnslothAI.Unsloth -e
Text
winget upgrade --id UnslothAI.Unsloth -e
Text
unsloth run --model unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0 \
    --reasoning-effort medium
BASH
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
BASH
pip install -U "huggingface_hub[cli]"
hf download unsloth/Qwen3.8-27B-GGUF \
    --local-dir unsloth/Qwen3.8-27B-GGUF \
    --include "*UD-Q4_K_XL*" # Use "*UD-Q3_K_XL*" for 3-bit
Text
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_XL.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0
Text
uv venv unsloth-nvfp4-env --python 3.13
source unsloth-nvfp4-env/bin/activate
uv pip install "vllm>=0.25.0" "flashinfer-python>=0.6.13" "nvidia-cutlass-dsl>=4.5.2" \
    --torch-backend=auto
Text
vllm serve unsloth/Qwen3.8-27B-NVFP4
Text
vllm serve unsloth/Qwen3.8-27B-NVFP4
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'
BASH
sudo apt-get update
sudo apt-get install -y ffmpeg
Text
uv pip install sglang sglang-kernel \
  --extra-index-url https://sgl-project.github.io/whl/cu130/ \
  --extra-index-url https://download.pytorch.org/whl/cu130 \
  --index-strategy unsafe-best-match
Text
sglang serve unsloth/Qwen3.8-27B-NVFP4
Text
sglang serve unsloth/Qwen3.8-27B-NVFP4 --speculative-algorithm EAGLE \
    --speculative-num-steps 3 --speculative-eagle-topk 1  --speculative-num-draft-tokens 4
BASH
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone --branch iq1-narrow https://github.com/unslothai/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
BASH
pip install -U "huggingface_hub[cli]"
hf download unsloth/Qwen3.8-2.4T-A95B-GGUF \
    --local-dir unsloth/Qwen3.8-2.4T-A95B-GGUF \
    --include "*Q1_0*" # Use "*IQ2_XXS*" for 2-bit
Text
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.8-2.4T-A95B-GGUF/UD-Q1_0/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0
BASH
pip install -U "huggingface_hub[cli]"
hf download unsloth/Qwen3.8-2.4T-A95B-GGUF \
    --local-dir unsloth/Qwen3.8-2.4T-A95B-GGFF \
    --include "*IQ1_S*" # Use "*IQ2_XXS*" for 2-bit
Text
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.8-2.4T-A95B-GGUF/UD-IQ1_S/Qwen3.8-2.4T-A95B-UD-IQ1_S-00001-of-00012.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0