Open-Source Lightreason Engine Accelerates Local Qwen Inference on RTX 4090 GPUs

A new open-source local inference runtime named Lightreason has demonstrated significant generation speedups for Qwen language models running on consumer hardware, according to benchmark documentation and project files published on github.com (https://github.com/me-farza/lightreason/tree/main).

Designed to operate on a single desktop Nvidia GeForce RTX 4090 with 24 gigabytes of video memory, the project combines graph execution caching, speculative decoding, and host-memory streaming to accelerate dense and mixture-of-experts architectures while retaining the models' original BF16 precision.

The benchmarks evaluate two specific architectures: the dense 3.09-billion-parameter Qwen2.5-3B-Instruct model and the 34.66-billion-parameter Qwen3.6-35B-A3B mixture-of-experts model. While the smaller 3B model fits completely within the GPU's 24 GiB onboard frame buffer, the 35B model's text weights occupy 64.56 GiB, forcing the runtime to stream expert weights between system RAM and the GPU over PCIe during generation.

For standalone runs of Qwen2.5-3B without auxiliary models, the project's target-only engine, called Lightreason Exact, increased decode throughput from 63.0 to 139.6 tokens per second across a standardized six-prompt suite. That reflects a 2.22-fold increase in raw decode rate and a 2.19-fold median improvement in paired warmed request latency compared to baseline Hugging Face Transformers execution. The speedup comes from capturing and replaying recurring execution paths via Nvidia CUDA Graphs, packing key-value caches into preallocated capacity buckets, fusing compatible BF16 projection kernels, and bounding vocabulary rows to screen out candidates before scoring survivors against original output weights.

Throughput scaled higher when speculative drafting was introduced. Integrating AMD's compiled PARD2 draft architecture alongside a custom auxiliary adapter, a tree verifier configuration reached 525.2 tokens per second on a complete weather dashboard generation task, up from 58.3 tokens per second on the reference runner. On a capped 128-token Fibonacci coding test, decode throughput reached 724.4 tokens per second, representing an 11.40-fold decode speedup and a 10.79-fold warmed request latency gain over Hugging Face. The project repository notes that reproducing these peak speculative figures requires a separately trained lab adapter checkpoint that is not bundled with the source release.

To support the 34.66-billion-parameter Qwen3.6-35B-A3B model—which routes each token through eight of 256 available experts per layer—Lightreason builds on FlashML's FreeToken hybrid CPU-GPU executor and Inco AI's DFlash2 speculative drafter. The pipeline maintains active experts in a 15-GiB GPU cache while routing remaining operations across system RAM and CPU threads.

Lightreason extends that hybrid execution model by introducing selected-row prefilling, AVX2 instruction reuse across repeated expert routes, lossless exponent-based host weight packing across 34 layers, and an 880-mebibyte GPU replica tier. This replica arena holds 187 packed experts compared to 146 raw uncompressed experts in the same memory budget, decompressing weights back into exact BF16 representations directly in hardware cache slots.

On a 768-token Python deduplication benchmark, Lightreason's packed replica setup delivered 51.08 tokens per second, up from 7.93 tokens per second for an initial scheduled offload baseline and 32.88 tokens per second for stock FreeToken execution. Developer documentation notes that host-to-device weight transfers over PCIe remain the primary performance barrier, consuming roughly 86 milliseconds per pass during multi-token verification and capping practical throughput around 50 to 54 tokens per second.

The repository includes an interactive local interface called Studio for testing prompt latencies and prototyping single-file web applications. Original Lightreason source code and documentation are released under the zero-clause BSD (0BSD) license, while model weights, FreeToken patches, and upstream draft checkpoints remain governed by their respective third-party licenses.