Why routing LLM calls is harder than it looks (lessons from building ai-gateway)

Most apps I’ve worked on treat LLMs in a very simple way: You pick a model → send every request to it → hope for the best. At first, that works. But over time I kept running into the same problems: simple queries hitting expensive models So I started building a small LLM routing layer that sits in f