Hightouch Benchmarks Hybrid Model Routing for Customer Identity Resolution

Pairing lightweight decision models with frontier large language models can match the auditing accuracy of top-tier systems at more than four times lower cost, according to benchmark findings published by data integration platform Hightouch on its blog (https://hightouch.com/blog/llm-judges-identity-resolution).

The tests examined automated evaluation for identity resolution, the core data engineering task of linking disparate customer records to unique individuals. The primary failure mode in identity systems is over-merging, where records from different people are erroneously collapsed into a single profile. Because live identity graphs lack ground-truth verification labels, data teams typically audit merge quality using manual human sampling—a bottleneck that averages only about 50 profiles reviewed per hour.

To evaluate whether artificial intelligence can replace manual auditing, Hightouch created a synthetic evaluation set of 1,000 identities using pseudopeople, a demographic simulation tool built by the Institute for Health Metrics and Evaluation at the University of Washington. The test set contained 500 properly merged identities injected with synthetic data noise—such as misspellings, flipped birth dates, and name variations—alongside 500 over-merged records. Hightouch evaluated frontier models including Anthropic's Claude Opus 5.5, OpenAI's GPT-6 Sol, and Google's Gemini 3.1 Pro against two specialized decision models, Jev and OpenAI's GPT-6 Luna, which output 0-to-1 confidence scores.

On baseline evaluations, Claude Opus 5.5 led with 99.0% accuracy at an average speed of 2.2 seconds per record and a cost of $3.86 per 1,000 identities. GPT-6 Sol and Gemini 3.1 Pro tied at 98.3% accuracy, costing $2.50 and $4.48 per 1,000 identities, respectively. By comparison, standalone decision models proved dramatically faster and cheaper when tuned with optimized confidence cutoffs: GPT-6 Luna achieved 97.3% accuracy at $0.10 per 1,000 identities (0.17 seconds each), while Jev reached 95.7% accuracy at $0.05 per 1,000 identities (0.15 seconds each).

To test model resilience, Hightouch assembled a harder benchmark populated exclusively with deceptive edge cases, including records of family members sharing names or addresses, twins, and discordant birth dates caused by typographical errors. On this challenging test, Claude Opus 5.5 dropped to 95.1% accuracy ($5.39 per 1,000 identities), while Jev alone fell to 89.1% and GPT-6 Luna declined to 88.8%.

Crucially, the benchmarking revealed that while Luna posted higher out-of-the-box accuracy on easy data, Jev demonstrated superior confidence calibration. Jev's classification errors clustered near the 50% midpoint, meaning the model showed uncertainty when making mistakes. Conversely, GPT-6 Luna displayed extreme overconfidence, frequently assigning 90% to 100% certainty to incorrect classifications on difficult records.

Hightouch capitalized on Jev's calibration by designing a tiered evaluation pipeline: Jev scores all records first, automatically escalating cases with confidence scores between 30% and 90% to Claude Opus 5.5. On the initial dataset, this hybrid setup reached 98.8% accuracy at $0.88 per 1,000 identities—cutting auditing costs by more than 75% compared to running Opus alone. On the harder test set, the hybrid Jev-Opus pipeline achieved 95.3% accuracy at $2.94 per 1,000 identities, edging out standalone Claude Opus 5.5 while reducing costs by roughly 45%.

Because Jev's confidence systematically degrades on ambiguous inputs, the architecture dynamically routes more volume to the expensive frontier model on complex workloads without requiring manual threshold adjustments. Hightouch noted that future research will examine how the hybrid approach handles under-merging and whether open-weight decision models can substitute for proprietary APIs.