How We Choose the Right AI for Each Feature
When you design an AI-based feature, the question "which model do I use" seems like something you should only have to answer once, at the start. In practice, that isn't how it went for NovLore: over time we realized that different tasks in the product had needs different enough from each other that a single provider, however good, couldn't serve them all well. This article explains why, and how the system that came out of it works today.
The problem that drove the decision
At first, Spark — NovLore's writing assistant — held two different jobs together in a single channel: the conversation with the user, and generating the work's dashboard (characters, places, plot) from that conversation. They seemed related, but they had opposite needs: conversation needs to respond fast, while generating consistent structured data needs to take the time to reason it through. A single provider, optimized for one of the two needs, systematically penalized the other.
The fix wasn't finding a provider that did both things well. It was to stop treating them as the same task.
Six domains, not one
Today every AI feature in NovLore declares its own domain — correction, rewriting, chat, structure, chapter, transcription — and each domain can have a different first-choice provider. Spark's chat, for example, uses a model chosen for response speed; generating a whole chapter, which the user launches knowing they'll have to wait, instead uses a model chosen for prose quality on a long task.
| Domain | What it does | What matters most |
|---|---|---|
| Chat | conversation with the assistant | response speed |
| Correction | typos and grammatical consistency | precision, low cost |
| Rewriting | editing a selected passage | stylistic quality |
| Structure | structured data on characters/plot | format reliability |
| Chapter | generating a whole chapter | quality on long text |
| Transcription | built but not yet active | — |
None of these choices is set in stone: an admin panel lets us change the provider and backup for each domain without touching the code, because the relative cost and quality of models shift over time faster than software does.
Why seven providers instead of one "good enough"
With a single provider, every outage — and it happens to everyone, even the best — becomes an outage for the whole product. The obvious fix is having a backup; the less obvious but sturdier one is that the backup can't run through the same channel as the first choice. If the first choice and the backup are two different routes to the same underlying provider, an outage at that provider takes both down together.
That's why, when a domain's first choice is an aggregator that gives access to several models with one key, the backup deliberately goes to an independent, direct provider. It's real redundancy, not just redundancy on paper.
What the user sees when the backup kicks in
Nothing, if the system works as it should. If the first-choice provider for a domain doesn't respond, the switch to the backup is transparent: the request still succeeds, and only an internal technical log records that the second choice was the one that answered. On streaming — responses that appear word by word, as in chat — the backup can only kick in before the first piece of text arrives: once a response has started, switching providers midway would stitch together text from two different styles, which is worse than waiting a few extra seconds.
The backup's transparency isn't a technical detail for infrastructure enthusiasts: it's the difference between a user who notices an occasional slowdown and a user who sees an error while writing their novel.
The router as the single point of passage
For this system to hold up over time, every AI feature in the product — including the ones with structured output and the ones with streaming — passes through a single internal router, never through providers directly. It's a choice that costs discipline (no shortcuts, not even for a "temporary" feature), but it pays off in a specific way: when a provider changes its terms, or a new one worth adding shows up, the change happens in one place, not across ten different endpoints written at ten different times.
Frequently asked questions
Why not always use the most powerful model available?
Because "more powerful" isn't the same thing as "better suited to the task." A model that excels at long, creative text can be slow and expensive for a chat that needs to respond in a second; using it everywhere would make even the part of the product that should feel instant seem slow.
Can the user choose which AI to use?
Not directly, message by message: the choice is made per domain by whoever administers the product, based on quality and reliability observed over time. It's a deliberate choice — leaving it to the user would push onto them a technical decision that requires continuously monitoring seven different providers.
What happens if the backup doesn't respond either?
The operation fails with an explicit error, and in the case of a credit-based action, the charge is automatically reversed onto the same batch it was taken from: the user doesn't pay for a request that got no response.
Does this system slow down building new AI features?
In the short term, yes, slightly: adding a feature means going through the router instead of calling an API directly. In the medium term it's the opposite, because every new feature inherits error handling, the backup, and cost tracking for free, already built once.