Production AI integration: a practical launch checklist

A support assistant can produce an excellent answer in a demonstration and still be unsuitable for release. The missing work appears when a document is stale, an account lacks permission, or the model times out after the user has already submitted a request.

For a concrete example, consider an assistant that drafts replies to customer questions using an internal product knowledge base. The proposed first release creates a draft for a support agent to review. It cannot send messages or change customer accounts. This is an illustrative design, not a report of a deployed AsyncX client system.

Put the task and its boundaries in writing

Specify the input, the expected output, and the actions the application is allowed to take. "Help support" is too broad to evaluate. "Draft a reply using approved documentation, with links to the supporting passages" gives the team something to inspect.

Decide what happens when the documentation contains no answer or two sources disagree. For this assistant, we would return a short explanation of the missing evidence and leave the ticket with the agent. A plausible invented answer would be a failure, even if its tone were excellent.

Separate advice from execution. A draft-only feature has a different release risk from a system that issues refunds. Expanding its authority later should require new tests and a deliberate permission change.

Build an evaluation set before polishing prompts

OpenAI's evaluation guidance recommends task-specific tests that reflect production use, continuous evaluation, and human review to calibrate automated scoring. It warns against judging a system only by whether outputs seem plausible. These are evaluation principles; they do not require a particular hosted evaluation product. OpenAI: evaluation best practices.

For the support assistant, we would assemble permissioned examples with expected behavior:

Test case	Expected behavior	Release concern
Current documentation answers the question	Draft an accurate answer with a supporting source	Answer quality
No approved source contains the answer	Explain the gap and hand back to the agent	Unsupported claims
Two documents disagree	Surface the conflict instead of silently choosing	Source handling
A user requests another customer's details	Deny access through application authorization	Data isolation
A retrieved document contains hostile instructions	Treat the content as evidence, without granting authority	Prompt injection
The model or retrieval service fails	Preserve the ticket and offer the normal workflow	Operational recovery

Agree which failures block release. Keep a separate set of examples for evaluating changes so repeated prompt tuning does not turn the test into a memorized demonstration. Add new failure cases from production only after applying the appropriate privacy controls.

A single average score can hide an unacceptable category. Review access violations separately from stylistic issues. A slightly awkward sentence and an exposed customer record must not cancel each other out in a combined quality number.

Enforce permissions outside the prompt

OWASP describes both direct prompt injection and indirect attacks through external content. Its defenses include least privilege, validation in code, and human approval for high-risk actions. A system prompt is one control, not a substitute for authorization. OWASP: prompt injection.

For our example, retrieval must use the requesting agent's permitted document scope before content reaches the model. The application must also enforce authorization when a tool runs. A model-generated account identifier is an untrusted parameter, even if it appears in valid JSON.

Keep service credentials out of model-visible context. Limit tools to the operations needed for the feature. Test an expired session, a removed team member, and a request that changes the target customer midway through a conversation.

The same boundary applies to output. Validate structured fields before using them. Render generated text safely. If links are permitted, apply the application's link policy before making them actionable.

Test the path after a partial failure

A timeout does not prove that a remote action never happened. AWS's guidance on retries explains why operations with side effects need idempotency before they can be retried safely. AWS: timeouts, retries, and backoff with jitter.

Our draft-only assistant can offer a retry without duplicating a customer-facing message because sending is outside its authority. If sending is added later, the application needs a durable operation identifier and a way to determine whether the send already completed.

Test failures after partial output, not only before the first token. Preserve useful text with an incomplete label when appropriate. Give the user a clear way to dismiss, retry, or continue manually. Cancellation should stop downstream work where supported, and late responses must not overwrite a newer request.

Set a latency and cost budget for the complete task

Measure from the user's action to useful output, and separately to completion. Include authentication, retrieval, model calls, and any follow-up generation. A fast model can sit inside a slow application.

Estimate cost per completed task using observed token usage, tool calls, retries, and review work. Track abandoned requests too. Avoid inferring production cost from a single short demo prompt.

Our AI response latency guide explains how to separate useful response time from optional work. A streaming interface can expose progress, but a successful-looking partial answer still needs a truthful completion state.

Make the first release easy to stop

Start with a defined audience and an operator who can inspect failures. Record the model and prompt version alongside the application release so a regression can be investigated. Limit retained content to what is needed and permitted; metadata and redacted examples can often answer operational questions.

For this assistant, a release review should require a passing permission test suite, acceptable results on the agreed evaluation set, a tested manual fallback, and a named owner for incidents. Use a feature control that disables the assistant while leaving ordinary ticket handling available.

Write down the stop conditions before launch. An authorization failure should trigger immediate containment. A decline in draft acceptance may call for a quality investigation. Different failures need different responses, and the person on call should not have to invent that policy during an incident.

The final rehearsal is simple: turn the AI service off and complete a support ticket. If the team cannot do that, the fallback is still a plan rather than a working part of the product.