Key takeaways
- List changing inputs and their authoritative sources before selecting a model.
- Compare retrieval, written instructions, ordinary software and training against the specific task.
- Test changed records, deleted documents and access restrictions as well as normal answers.
A quotation assistant may need today’s price, a current product specification and a discount policy that has not changed in a year. Those inputs come from different places and need different update schedules. Listing them separately gives an engineering team a more useful starting point than choosing a model from a general benchmark.
For each input, identify the authoritative source, who maintains it and how quickly a change must reach the assistant. Ask the person responsible for prices how corrections are published. Ask the policy owner how a new version replaces an old one. Record exceptions, including temporary offers and rules with future effective dates.
Look up information that must reflect a current record
Retrieval means supplying relevant information to a model when it answers a request. A system might search approved documents or query a database, then use the returned material to draft its response. Patrick Lewis and colleagues’ 2020 paper on retrieval-augmented generation combined a language model with a searchable document collection. This approach separates some reference information from facts stored in model parameters during training.
A direct query to an authorized pricing system can provide a current price. A document search might provide a policy passage. Decide which lookup is appropriate for each input, and specify what happens if the source is unavailable. For a price required to complete a quote, stopping for staff review may be preferable to using an older cached figure.
Retrieval also needs maintenance. A file may change before a search index refreshes, an obsolete policy may remain searchable or an employee may lack permission to read a document. Test updates, deletion and access restrictions alongside answer quality.
Choose how to handle rules and examples
Fixed calculations can be implemented directly in software. A language model can extract quantities from a request while a calculation function applies an agreed formula. Test that extraction separately: correct arithmetic on a misread quantity still produces the wrong quote.
Examples and written instructions may be enough to teach the desired format for a response. Fine-tuning changes a model through additional training and is another option when repeated tests show a specific behavior needs improvement. Compare it with supplying examples in the request, using the same test cases. Include the cost of collecting and maintaining training examples.
Consider a searchable handbook when users need to see where an answer came from, even if it changes infrequently. Compare freshness, accuracy, access controls and maintenance costs when choosing how to provide that information.
An information inventory for a quotation assistant
| Input | Proposed source | Change to test |
|---|---|---|
| Price | Authorized pricing system | A price correction made after a draft was started. |
| Product specification | Approved product records | A discontinued item or replacement specification. |
| Discount policy | Versioned policy document or coded rules | A new effective date and an exception requiring approval. |
| Customer terms | Customer-specific contract records | A user who cannot access another customer’s agreement. |
Illustrative design questions.
Compare complete systems on the same tasks
Run a proposed assistant on ordinary requests and on cases involving recently changed records. Keep the expected answer, source version and time of retrieval for each. If an answer is wrong, identify whether the lookup missed information, supplied an old record or was misread by the model.
Measure cost per completed, acceptable task. Include searches, model calls, retries and staff review, along with software and maintenance costs. A cheaper model may need more correction; a larger one may provide no useful improvement on this task. The comparison needs those observed results before it can support either choice.
Assign an owner to every source that needs updating. Give that person a test request to run after a change, such as asking for a newly corrected price or a policy with a new effective date. Save failures with enough detail for the engineer to reproduce them.
