Neuralwatt Study Finds AI Client Harnesses Drive Substantial Swings in GPU Energy Use

Client-side agent harnesses exert a substantial influence on GPU energy consumption and inference efficiency, according to production data and laboratory testing published on Hacker News by cloud inference provider Neuralwatt (https://neuralwatt.com/news/why-harnesses-matter-for-energy-efficiency).

Chad Gibson, co-founder and chief executive of Neuralwatt, noted that agentic workloads produce heavily asymmetric traffic where input volume dwarfs generation. Typical coding-agent interactions submit 100,000 tokens or more of context to generate a few hundred tokens of output. Serving this traffic efficiently relies on prompt caching via key-value (KV) memory to avoid recalculating unchanged text. Because caching requires an exact prefix match, any edit inserted early in the prompt by a client harness forces the GPU to recompute the prompt from the first point of difference.

Analyzing traffic across hundreds of client applications from September 8 to October 7, 2026, Neuralwatt found that OpenCode accounted for roughly one in four customers on its platform. Small variations in cache hit rates between tools translated into significant computational differences: a client running at a 92 percent caching rate sent 2.7 times as much uncached prefill work as a tool running at a 97 percent rate. Deepseek-harness broke cache on roughly one turn in 100, compared with about three turns in 100 for OpenCode and Pi.

Direct GPU energy measurements recorded between October 1 and October 7, 2026, for 50,000-token prompts showed that mostly uncached requests consumed roughly twice the energy of fully cached requests on identical models, rising to two and a half times on GLM-5.3. Gibson noted that wasted prefill also causes GPU resource contention, slowing down generation for other concurrent workloads on the same server.

In extended sessions evaluated from October 6 to October 13, 2026, on DeepSeek V4.1 Flash, GLM-5.3, and GLM-5.3 Flash, Neuralwatt observed severe cache penalty spikes at deep context levels. Beyond 128,000 tokens of context, about three in every 1,000 turns re-read more than half the prompt from scratch. These instances accounted for 75 percent of all re-read tokens at that depth, reprocessing an average of 230,000 tokens per break—equivalent to the prefill work of roughly 600 standard turns adding a median of 370 new tokens.

Controlled proxy testing on DeepSeek V4.1 Flash and GLM-5.3 using OpenCode 1.18.34, OpenCode 2.0.29, Pi 1.0.4, omp 18.6.3, and Kilo Code 7.8.3 showed that 88 percent of 539 follow-up requests maintained exact prefix appends. All 65 cache breaks stemmed from client harness behavior, such as reordering tools, changing reasoning effort settings, updating skills, or compacting history. In production metadata, altering tool lists or reasoning effort increased the likelihood of a cache break by more than 20 times and accounted for 19 percent of all re-read tokens on tracked models.

Neuralwatt has rolled out a Session Insights feature in its dashboard allowing users to track per-turn cache hits, latency, and measured GPU energy. Gibson, who spent more than 20 years at Microsoft before co-founding Neuralwatt in 2024, said the company plans to submit fixes upstream to open-source agent projects while developing server-side prefix tolerance techniques.