When the algorithm beats the AI
- The problem
- Inbound request routing in a B2B support or ops platform — thousands of items per minute at peak. Looks like an LLM job at first glance.
- The options
- (a) An LLM classifier with a structured output schema, (b) a small fine-tuned encoder model, (c) deterministic rules plus a lookup table driven by measured traffic distribution.
- My default
- I sample a week of production traffic before choosing. In most B2B ops pipelines, a dozen rules cover the large majority of volume and the remainder clusters into a handful of tenant-tier × product-line buckets. Rules plus lookup for the hot path; a cheap small-model classifier for the long tail; no LLM on the live path.
- What it buys you
- Millisecond-range latency, near-zero variable cost, a classifier an on-call engineer can debug at 3 a.m. without looking up an API key. The trade: harder to extend to categories that don’t exist yet. For constrained inputs, that’s a good trade. Reach for an LLM when the input space is genuinely unbounded — not by default.