From ML Models to AI Systems: What Changes in Production?

• A model is one part of the service
• Follow one customer question
• The data becomes a continuing responsibility
• Evaluation expands to the whole task
• Speed and cost belong to the request
• Decide what happens when the system cannot answer
• A release starts an operating loop
• What to build first
• References

A model gives a convincing answer in a demo. Then a customer asks the same kind of question in the live product, and the answer is wrong. The model has not changed. The information around it has.

Imagine a support assistant answering this question: “Can I return an item I bought 45 days ago?” The store has extended its return window from 30 to 60 days, but the assistant still reads the old policy. It explains that outdated policy clearly and confidently.

This is a fictional example, but it exposes the question this article explores: what has to work around a model before people can depend on its output?

My Introduction to Machine Learning covered how models learn and generalize. Here, we follow the next step: connecting a model to real data, real users, and an ongoing service. You do not need experience running AI infrastructure to follow the example.

A model is one part of the service

A machine learning model uses patterns learned from data to produce an output: a category, a numerical prediction, or generated text. Using a trained model on an input is called inference.

An AI system includes that model and the surrounding work needed to deliver a useful result. It obtains inputs, prepares them, calls the model, applies application rules, and returns something a person or another program can use. People also need to operate and maintain it.

For a support ticket classifier, the model might predict “returns.” The system still needs to read the message, route it to the right queue, handle unavailable services, and let someone correct a mistake. A correct label does not help if the ticket never reaches the support team.

These responsibilities existed before large language models. Sculley and colleagues described how data dependencies, interacting components, and feedback loops create maintenance problems in production ML systems in 2015. Their argument concerns the system around the algorithm, as well as the algorithm itself. Sculley et al., 2015

A large language model, or LLM, can make the boundary less obvious because one model can perform many tasks through text instructions. Yet it still needs a surrounding application. A model that writes an answer does not, by that ability alone, know which customer records it may read.

Berkeley researchers use the term compound AI system for an application in which models work with other components, such as search or external tools. A system can follow a fixed sequence; it does not need an autonomous agent or several models. Zaharia et al., 2024

The title’s move from “ML models” to “AI systems” is therefore a change in what we design and evaluate. Traditional ML also runs inside systems.

Follow one customer question

Return to the fictional store. For this example, assume that the new 60-day policy applies to this order and that we are explaining the time window, not checking every condition for a refund.

A useful assistant needs information from two places: the customer’s order record and the applicable policy. Its language model supplies the ability to interpret and explain that information.

One possible request path is:

1. Check access. The application verifies who the customer is and whether they may access the requested order.
2. Read the order. It obtains the purchase date and relevant order details from the store’s system.
3. Find the policy. It retrieves the policy that applies to this order, including its effective date and relevant conditions.
4. Prepare the model input. It combines the question, the permitted order information, the policy passages, and instructions for answering.
5. Draft and check the answer. The model writes a response. Application code checks things it can verify directly, such as the response format and whether the cited document identifiers were actually retrieved.
6. Respond or hand off. The application returns the answer, requests missing information, or sends the case to support.

The combination of retrieving information and using it to generate a response is called retrieval-augmented generation, or RAG. Lewis and colleagues studied a model architecture that combined a retriever with a text generator. Today’s applications may implement that combination differently; the useful idea here is that the answer can use information fetched at request time. Lewis et al., 2020

The diagram shows this example’s successful request path. Each arrow means “the next step”; the model occupies one step in the larger process.

flowchart TD
    A["Customer question"] --> B["Check access; read order"]
    B --> C["Find applicable policy"]
    C --> D["Prepare model input"]
    D --> E["Model drafts explanation"]
    E --> F["Check format and source IDs"]
    F --> G["Return answer with source"]

This illustrative path assumes access is permitted, required information is available, and response checks pass. Otherwise the application stops, clarifies, or hands off as described above. Document ingestion, logs, and service failures are discussed below.

A format check cannot prove that an answer is true. A valid citation can still accompany an incorrect interpretation. In this example, application code can calculate the order’s age; checking a complicated policy interpretation may require a person. Those are different kinds of checks, and calling both “validation” should not hide the difference.

Now the original failure becomes easier to locate. The model may have followed the supplied text accurately. The system failed because it supplied an obsolete policy.

The data becomes a continuing responsibility

During development, a dataset is often a prepared snapshot. A live application needs a process that keeps its inputs usable as the world changes.

For the assistant, policy documents must be collected, parsed into usable text, and made searchable. The searchable collection is often called an index. If the store edits a page but the index is never refreshed, the assistant can keep retrieving the previous version.

A refresh alone is not enough if both versions remain searchable without dates or applicability information. The application needs to distinguish “this policy is newer” from “this policy applies to this order.”

This suggests a concrete debugging order. If the answer says 30 days, inspect the retrieved passage first. If it contains the old rule, investigate document ingestion and policy selection. If it contains the correct 60-day rule but the answer still says 30, investigate how the input was assembled and how the model used it. Replacing the model addresses neither diagnosis automatically.

There is a related problem in predictive ML: live inputs may differ from the examples used for development. A classifier trained on formal email messages might later receive short, informal chat messages. This is a distribution shift: the kinds of inputs, outputs, or relationships between them differ.

The WILDS benchmark documents performance drops under real-world shifts across settings such as hospitals and camera locations. It supports the need to test beyond a convenient development dataset; it does not measure the support assistant in this article. Koh et al., 2021

A change in input patterns is a reason to investigate. It does not establish that accuracy fell or that retraining is the right repair. A broken parser, an outdated index, and a model that no longer fits its task require different interventions.

Evaluation expands to the whole task

A model test asks whether the model performs well on selected examples. A system test asks whether the entire application delivers the intended result under relevant conditions.

Breck and colleagues’ ML Test Score organizes production checks across data, model development, infrastructure, and monitoring. It also calls for checking whether offline measurements relate to actual user-facing outcomes. Its specific rubric predates modern LLM assistants, but the broader testing distinction remains useful. Breck et al., 2017

For our assistant, the test set should express what a satisfactory response means. It need not require one exact sentence. For the 45-day order, an acceptable answer must use the applicable 60-day policy, explain the time window correctly, cite the source, and avoid promising a refund when other conditions have not been checked.

Different checks answer different questions:

Check	Question in the support example
Access and data	Did the application use only records this customer may access?
Retrieval	Did it find the applicable policy and the passage needed to answer?
Answer quality	Does the explanation match the order information and the policy?
Failure behavior	Does missing information lead to clarification or a useful handoff?
User outcome	Did the customer get a correct next step, without an unsupported promise?

Keep tests for individual components because they help locate failures. Also run the complete request path because individually working parts can interact incorrectly.

Include ordinary questions and cases that challenge the assumptions: an old order, an ambiguous purchase date, a missing policy, a question in another supported language, and a request for somebody else’s order. Review those groups separately so a large number of easy questions does not hide a weak area.

RAG also needs tests for how the model uses retrieved text. In Lost in the Middle, Liu and colleagues found that answer quality depended on where relevant information appeared in the context for the models and tasks they studied. This is a reason to test document selection and arrangement, rather than assume that adding more text guarantees a better answer. It is not a universal result about every current model. Liu et al., 2024

For a small initial test set, people can inspect answers against the source passages. Automated checks can then take over clearly specified parts. Any automated judge used for harder judgments should itself be checked against human-reviewed examples; otherwise the evaluation introduces another unverified decision.

Speed and cost belong to the request

The customer experiences the time from asking a question to receiving a usable answer. The model’s processing time is only part of that wait. Authentication, order lookup, policy retrieval, queueing, and response checks also take time.

For steps that run one after another, their waiting times accumulate. If some steps run concurrently, the timing depends on which results later steps must wait for. Measuring only the model call can therefore miss the part that is actually slowing the product down.

Track the distribution of response times as well as an average. The 95th percentile, or p95, is a response-time threshold at or below which roughly 95% of the measured requests fall. It helps reveal delays that an average can conceal. Google’s SRE monitoring guidance explains why the slow end of a latency distribution matters, alongside traffic, errors, and resource saturation. Ewaschuk, 2016

For a streaming assistant, distinguish the first visible text from the completed answer. Starting to display words quickly does not tell us when the customer receives the information they need.

Cost also extends beyond one model invocation. Count the paid or provisioned work used by the request: model calls, retrieval, retries, and relevant operating costs. Track human handling separately or include it with an explicit accounting rule.

Suppose an assistant makes cheaper individual calls but often needs several attempts and a support handoff. A comparison based only on cost per call misses that work. A useful additional measure is total cost over a period divided by successfully completed tasks, with “success” defined in advance. This is an accounting choice for the application, not a benchmark result.

The practical goal is to choose an acceptable combination of answer quality, response time, and cost for the task. Those budgets apply across the components, a design issue also discussed in the Berkeley article on compound systems. Zaharia et al., 2024

Decide what happens when the system cannot answer

A missing policy, a slow order service, and an unclear question are predictable situations. The product needs a response for each.

If the order service is unavailable, the assistant should explain that it cannot currently check the order and offer a next step. It should not fill the gap with an invented purchase date. If the policy is ambiguous, the interface can show the relevant source and hand the case to support.

Set limits on waiting and retries. A timeout limits how long the application waits for an operation; a retry makes another attempt. The appropriate limits depend on the operation and the user’s task. Retrying an information lookup and retrying an action that changes an order need different handling. If a refund was processed but its confirmation was lost, repeating the request could issue it twice unless the service recognizes it as the same operation.

This becomes especially important when an assistant can take actions. Explaining a return policy and issuing a refund are different capabilities. In our example design, the application would enforce permissions and refund rules independently of generated text, and require the appropriate approval before an action.

Retrieved content creates another failure path. A document can contain text that tries to instruct the model rather than inform it. This is indirect prompt injection. Greshake and colleagues demonstrated attacks that placed malicious instructions in material consumed by LLM applications. Greshake et al., 2023

The design implication is to treat retrieved text as untrusted input and limit what the application can do with model output. Access controls belong before protected information reaches the model; action permissions belong at the tool or service boundary. A prompt that says “follow the rules” is not an access-control mechanism.

A release starts an operating loop

Once people use the assistant, its operators need to understand both whether it is available and whether it is helping.

Operational measurements reveal failed calls, slow requests, and overloaded resources. Content review answers a different question: did the customer receive correct guidance? A service can return a technically successful response containing the wrong answer.

For the fictional 30-day answer, a useful request record would identify the model version, instruction version, retrieved policy IDs and versions, timings, and outcome of the application checks. This lets an operator distinguish an obsolete document from a bad interpretation.

Record only what is needed for that investigation. Order details and customer messages may be sensitive; access restrictions and retention limits apply to diagnostic records too. The ML Test Score explicitly includes privacy controls for data pipelines and monitoring for changes in prediction quality. Breck et al., 2017

Version the parts that can change behavior. Depending on the application, these include the model, input preparation, instructions, retrieval configuration, document collection, and application rules. Keeping the same model does not mean the same system is running.

Before introducing a change broadly, evaluate it on the saved cases. Where the risk permits, use a canary release: expose a limited portion of live traffic to the change, compare it with the existing version, and decide whether to continue. Google’s SRE workbook describes this as a way to discover release problems while limiting exposure. Warner and Davidovič et al., 2018

Plan how to stop or reverse the change, and assign someone to respond when the checks fail. Restoring an older application version is useful only if its dependencies still work and its behavior remains acceptable. Restoring an obsolete return policy would not be a successful recovery.

Finally, decide how feedback enters future development. A customer’s positive rating is evidence about their experience, not proof that the policy answer was correct. If generated answers are later treated as training truth without checking them, mistakes can feed back into the system. Feedback loops are among the maintenance risks identified in the earlier technical-debt research. Sculley et al., 2015

What to build first

The example does not imply that every project needs a large platform. A batch classifier, a recommendation service, and an interactive assistant have different constraints. The amount of operational machinery should follow the task, consequences of failure, and scale.

For a first useful version, write down the intended outcome and trace one request from input to result. For each step, identify the required data, a meaningful failure, and the behavior the user should see. Then establish a small set of representative tests, observe the live service, and make changes reversible where possible.

In the support example, that means being able to explain why the assistant answered 30 or 60 days, detect when it uses the wrong policy, and give the customer a useful next step when the information is unavailable.

A good model provides a capability. A dependable AI system makes that capability useful under the conditions in which people actually use it.

References

Research papers, an expert perspective, and engineering guidance serve different purposes here. None of them is presented as a measured evaluation of the fictional support assistant. Links and publication details were checked on 12 September 2026.

1. D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28, 2015, pp. 2503–2511. Google Research record · Paper.

2. Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D. Sculley. The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. 2017 IEEE International Conference on Big Data, 2017, pp. 1123–1132. DOI: 10.1109/BigData.2017.8258038 · Author-hosted paper.

3. Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. The Shift from Models to Compound AI Systems. Berkeley Artificial Intelligence Research Blog, 18 February 2024. Expert perspective; not a peer-reviewed benchmark. Article.

4. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. arXiv:2005.11401 (linked revision: v4, 2021).

5. Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A Benchmark of in-the-Wild Distribution Shifts. Proceedings of the 38th International Conference on Machine Learning, PMLR 139, 2021, pp. 5637–5664. Published paper and metadata · arXiv:2012.07421.

6. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 2024, pp. 157–173. DOI: 10.1162/tacl_a_00638 · Published paper · arXiv:2307.03172 (preprint first posted in 2023).

7. Rob Ewaschuk; edited by Betsy Beyer. Monitoring Distributed Systems. Chapter 6 in Site Reliability Engineering: How Google Runs Production Systems, O’Reilly, 2016. Practitioner guidance. Chapter.

8. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. 2023. Citation here is to the arXiv preprint, v2, 5 May 2023. arXiv:2302.12173v2.

9. Alec Warner and Štěpán Davidovič, with Alex Hidalgo, Betsy Beyer, Kyle Smith, and Matt Duftler. Canarying Releases. Chapter 16 in The Site Reliability Workbook: Practical Ways to Implement SRE, O’Reilly, 2018. Practitioner guidance. Chapter.

AI assistance: AI tools assisted with source discovery, synthesis, drafting, and language editing. The support scenario and diagram are illustrative; no production benchmark or employer case study was conducted for this article.