Internal AI tool
LLM Routing System
An on-device LLM routing layer that classifies each prompt and decides whether the request should stay local or use a cloud model.
- My role
- Designer and implementation contributor
- Context
- Porsche Digital internship, Vehicle Solutions & Data
- Reading time
- 6 min read
01
One model is rarely the right answer for every task
Teams using LLMs face a recurring tradeoff. Local models can be private, fast, and inexpensive to run, but they do not handle every task equally well. Cloud models can offer stronger capability, but using them for simple work adds cost and sends more requests outside the local environment.
The project explored that decision as a routing problem. Instead of fixing every request to one provider, the system first examines the task and chooses an execution path.
02
A local decision before any external call
The router uses an on-device LLM to inspect the incoming prompt, identify the task type, and estimate its complexity. Routing logic then sends the request to a suitable local model or to a configured cloud provider when the task needs stronger capability.
The classification decision itself remains local. That detail matters because a router that sends every prompt to an external classification API has already lost much of the privacy and cost benefit it was supposed to create.
03
Keeping the routing logic understandable
Routing systems can become opaque quickly. Model confidence, task categories, fallback behavior, provider availability, and errors all affect the final choice. I kept the path explicit so a team could inspect why a request moved to a particular tier and adjust the policy as models changed.
The design also had to allow new local and cloud models without rebuilding the whole decision layer. The router owns selection; provider adapters own execution. That separation keeps experimentation possible without mixing provider details into every rule.
04
What the routing changed
My working estimate was that the router kept about 65 percent of requests on local models and reduced cloud AI spend by roughly 45 percent. These are my own operational estimates, not figures from a controlled finance study, so I present them with that qualification.
The more durable result was the decision layer itself. Simple tasks could stay on-device, stronger models remained available when needed, and the classification step did not require an external call. Internal providers, prompts, evaluation data, and deployment details remain confidential.
Outcomes
- 01A working local classification and routing approach for mixed local and cloud model access.
- 02An estimated 65 percent of requests routed locally and an estimated 45 percent reduction in cloud AI spend, based on my own operational assessment.
- 03A provider-independent structure that can evolve as model capability and availability change.
- 04A routing decision path that avoids an external API call before the execution choice is made.
What I learned
- 01Model routing is a policy problem as much as a classification problem.
- 02A local-first path only remains local-first if classification, logging, and fallback behavior respect the same boundary.
- 03Routing explanations matter because teams need to tune cost and quality decisions over time.