llama.cpp

Datasette-llm, the intermediate plugin that configures LLMs for other Datasette plugins, released version 0.1a11 with support for OpenAI-compatible endpoints. Users can now point Datasette at any service speaking the OpenAI pseudo-standard protocol, including local backends like LM Studio, Ollama and llama.cpp, to power a Datasette Agent entirely on local models.

llama.cpp and its underlying tensor library ggml have both shipped 1.0 releases, signaling API stability, with llama.cpp removing 35 deprecated functions from its public header. SGLang also announced its 1.0 at its summit, marking broader engineering maturity for local inference and high-performance serving stacks.

ggml has released v1.0.0, marking the first formal versioned milestone for the tensor library underpinning the llama.cpp ecosystem. Its core value is a dependency-free C/C++ implementation spanning x86, ARM, RISC-V and more, with 2- to 8-bit integer quantization plus MXFP4 and NVFP4 microscaling formats and zero runtime memory allocation. For local-inference and edge-deployment developers, this signals long-term ecosystem stability.

llama.cpp and its underlying tensor library ggml have officially reached version 1.0.0, establishing a semantic versioning scheme that flags breaking changes to the public C API. The project now supports over 150 model architecture variants and mainstream speculative decoding methods, backed by expanded self-hosted CI runners across NVIDIA, AMD, Qualcomm, Arm, IBM and Intel hardware.

Unsloth released Dynamic v3.1 quantizations for the Qwen3.8 family, including the 27B dense model and the 2.4T-A95B MoE variant, shipping GGUF weights across UD-Q4_K_XL down to IQ1_S bit-widths. The update pairs one-line installers with ready-made llama.cpp deployment commands, lowering the barrier to running very large MoE models locally.

llama.cpp has officially released v1.0.0, establishing a stable baseline 3.5 years after inception. Built on quantization, optimized backends and a portable C/C++ implementation, it runs multimodal LLMs from consumer hardware to enterprise servers, serving as a key foundation for local and private AI inference.

llama.cpp's tiled K-quant matmul previously shipped only AVX-512, AVX2 and AVX inner kernels, forcing Q3_K, Q2_K and IQ-family formats on Arm into a slow path that re-unpacked weights for every dot product. This contribution adds the missing Arm inner kernel built on the NEON SDOT instruction, lifting IQ3_M prompt processing on Neoverse N1 from 11.5 to 74.5 tokens/s (about 6.5x), with 3.3x to 4.9x gains on Neoverse V1 and V2.

A Reddit hobbyist ran the 2.8-trillion-parameter Moonshot Kimi K3 model in a GPU-free home lab using discontinued Intel Optane Persistent Memory 200 series and llama.cpp. With 1-bit quantization, disabling DAX so hot data benefits from DRAM caching lifted throughput from 0.85 to 1.5 tokens per second, while a 3-bit test reached only 0.75 tokens/s, taking about 18.5 hours for 50,000 tokens. The experiment shows trillion-scale models can run entirely from a memory hierarchy instead of GPU VRAM, though at far-from-practical speeds.

Liquid AI released LiquidCodec, a ~74M-parameter streaming audio codec that converts speech into tokens a language model can predict and back into audio, powering LFM3-Audio-2B and LFM3-Omni-2B. Its key innovation replaces residual vector quantization with grouped finite scalar quantization (GFSQ), producing 12 independent tokens per 80 ms frame that can be predicted in parallel, while the decoder predicts spectral coefficients directly and skips transposed-convolution upsampling. On a single CPU thread of an Apple M4 Max (FP32) it encodes at ~361x and decodes at ~87x real time, with an encode RTF of 0.0028 versus 0.0361 for Kyutai's Mimi, and posts the best ViSQOL and TTS UTMOSv2 across all 8 test sets; however, PESQ still trails NVIDIA's NanoCodec on 7 of 8 sets, and no bitrate or latency in milliseconds was published. Weights are on Hugging Face with GGUF builds for llama.cpp.

Huawei has open-sourced MACaQ, an internal model-agnostic calibrated quantizer that spans text, image, video and audio models across llama.cpp, vLLM, Diffusers and PyTorch backends. It claims to quantize a 122B-parameter model in 64 minutes using only 32.5 GB of device memory, and to match MR-GPTQ and DuQuant++ perplexity on Llama-3-8B with 11x less quantization time.