⬢github C++ · 645 ★ · pushed 7 h ago
kvmem/kvmem-llama.cpp
A fork of llama.cpp that adds tiered KV-cache memory: completed KV blocks live in host RAM, and a small bounded GPU window is filled with relevant retrieved blocks per step. This lets large-context models like Qwen3.8-27B run at full 256K context on a 16 GiB GPU with near-lossless quality, served via an OpenAI-compatible chat server.
Why now: Recently at the top of trending repos with a new v0.16.0-rc3 prebuilt release, plus benchmark comparisons against KV-cache streaming showing 32-33 tok/s decode on an RTX 5060 Ti.
Who it is for: Developers running long-context local LLM inference on consumer GPUs with limited VRAM.
Stars over our 11 snapshots: 645 to 645, since 1 h ago.
Where people talked about it
- ⬢github new repos, most starred just now
API: https://socialmediatrends-api.osmike.com/v1/repos/kvmem/kvmem-llama.cpp