Model-Agnostic Routing: Using Evaluations to Balance Latency, Cost, and Accuracy

How ALLTIPLY architected a model-agnostic routing gateway for a $320M commercial lending portfolio.
Model-Agnostic Routing: Using Evaluations to Balance Latency, Cost, and Accuracy
industry
Commercial Lending & FinTech
location
Solutions
Model Performance & Applied AI Lab
Problem
$42,000/month single-provider AI bill, 3.8-second underwriting wait time.
Solution
$6,800/month blended AI cost (84% savings), 420ms response time (89% faster), zero vendor outages.

Executive Summary

A commercial lending firm was spending over $42,000 a month on a single frontier model API, while loan officers complained about 4-second loading times on basic application screens.

ALLTIPLY designed an intelligent 3-tier routing gateway that sends routine data extraction to fast 8B small models and reserves expensive reasoning models for complex legal exceptions.

Business Results

  • Monthly AI Spend: Slashed by 84% (from $42k/mo to $6.8k/mo).
  • Underwriting Latency: Accelerated by 89% (from 3.8s to 420ms).
  • Vendor Reliability: 100% uptime with automated zero-downtime failover.