When an AI tool produces a disappointing result, it is tempting to conclude that the model is not capable enough. Sometimes that is the constraint. But before changing models or abandoning the idea, I think there is a more useful question to ask: have we built a system that gives the model what it needs to succeed, lets us see where it falls short, and provides a way to improve?
Andrew Ng's work is an excellent reference point for answering that question. His Agentic AI course at DeepLearning.AI connects the design of AI workflows with evaluation, error analysis, and practical improvement. What I especially appreciate about his teaching is the combination of technical depth and clarity. He makes complex ideas understandable without losing the details that matter when building something real.
This article starts with his framework, then looks at how I apply those ideas to tools and workflows in my own work. The central lesson applies whether the system serves an internal team or an external customer: generating an output is only part of the job. We also need to understand its quality, investigate failures, and use that evidence to make the next version better.
Andrew Ng's Four Agentic Design Patterns
Ng describes four patterns for building agentic AI: reflection, tool use, planning, and multi-agent collaboration. Together, they provide a useful vocabulary for understanding how an AI system can move beyond a single response and work through a task.
Reflection
Reflection introduces a review-and-revision step. Instead of treating its first output as finished, the system critiques that output and uses the feedback to produce another version.
For a report, that could mean generating a draft, reviewing it against specific requirements, and revising sections that are incomplete or unsupported. The review becomes more useful when it has something concrete to work against, such as source material, required sections, or examples of acceptable work.
The evaluation question is whether the revision actually improves the result. I would compare the initial and revised outputs against the same criteria, including whether the revision introduces new problems. A second draft should not receive a better score simply because it is longer or sounds more confident.
Tool Use
Tool use allows a model to interact with resources outside the model itself. That might involve searching for information, querying a database, executing code, or interacting with another application.
This changes what we need to evaluate. For a research tool, I would want to know whether it selected the right source, retrieved the relevant material, and used that material correctly. A successful connection to a database is not the same as finding the information needed to answer the question.
This distinction helps make failures more specific. If the output is missing evidence, the problem might be the search query, the material returned, or the interpretation of that material. Each suggests a different intervention.
Planning
Planning gives the model responsibility for breaking a task into steps and adapting those steps as it proceeds. In a fixed workflow, the developer defines the sequence. In a more dynamic workflow, the model makes some of those decisions based on the task and the results it encounters.
For a research assignment, a plan might involve identifying the question, locating relevant evidence, checking for gaps, and deciding how to organize the findings.
I would evaluate whether that plan addresses the actual request, includes necessary checks, and responds appropriately when information is missing. A system can execute its steps successfully while still failing the overall task because the plan left out something important.
Multi-Agent Collaboration
Multi-agent collaboration distributes responsibilities across specialized agents. One might gather information, another develop the output, and another review it. These agents can also use tools, plan their work, and revise their outputs.
For this arrangement, I would evaluate the handoffs as well as the individual contributions. Did the drafting agent receive the supporting evidence? Did the reviewer receive the original requirements? Did an important qualification disappear between stages?
I treat these four patterns as design options, not a checklist that every tool must satisfy. Adding another agent or another review step should address an observed problem. The evaluation should help determine whether the additional complexity is worthwhile.
Evaluation Tells Us What Works. Observability Helps Explain What Happened.
An important part of Ng's course is the attention given to evaluations, error analysis, and component-level testing. The design patterns describe ways to organize the work. Evaluation helps determine whether that organization is producing the intended result.
An evaluation is a structured assessment against defined criteria. For a presentation tool, those criteria might include required content, factual support, formatting, and suitability for the audience. An end-to-end evaluation examines the finished deliverable. A component-level evaluation isolates a particular part of the workflow, such as retrieval or slide generation.
Observability provides the records needed to investigate performance. Traces show what the system did, allowing a team to inspect a run rather than infer everything from the final answer. They also provide material for debugging and future evaluation cases.
For the systems I build, I want to preserve enough information to answer practical questions: what input did the system receive, which references did it access, what did its tools return, and what did it produce at the important stages?
Suppose a presentation is missing a required section. The source material may never have reached the model. The outline may have omitted the section. Or the content may have been generated correctly but lost during slide creation. The final presentation reveals the problem, but the intermediate records help locate it.
Different evaluation methods serve different purposes. Code-based checks can verify required fields or calculations. Model-based evaluations can assess outputs against a rubric. Human reviewers help establish the standard, resolve difficult judgments, and check whether automated scoring agrees with expert assessment.
The goal is to combine these methods around the work that matters, rather than ask one score to explain the entire system.
Applying This to a Research Presentation Tool
I encountered this directly while building a presentation tool to streamline the user research process.
The early output was not particularly good. It was missing components, and its formatting and content did not consistently match what we needed. The immediate reaction could have been that AI was not ready for the task.
Instead, I edited the presentation and placed the revised version somewhere the system could access it. That created a comparison between what the system generated and what I actually used.
The comparison made our expectations more concrete. We could examine what needed to be added, what needed to be reorganized, and how the presentation needed to change to fit our standards. It gave the system examples of our content and formatting preferences, rather than relying entirely on instructions written in advance.
Over time, we used those differences to refine the workflow and bring the output closer to what we wanted.
For me, the important lesson was not that every weak output can be fixed through configuration. It was that the first output did not give us enough evidence to conclude that the model was the limiting factor. We needed a way to inspect the gaps and address them systematically.
Turning the Difference Between "Generated" and "Used" Into Feedback
A related approach we have used is a shared feedback log.
The system generates a draft, and a person edits it into the version they actually use. The workflow retrieves both versions, compares them, and records the differences. Once a week, an automated AI review reads the log to identify recurring changes and nuances.
The log is also available to the people responsible for the work. Authorized team members can review the comparisons, correct the AI's interpretation, and add their own feedback.
This creates a useful combination of automated collection and human judgment. The system can capture changes without requiring someone to document every edit manually. People can then explain the decisions that a comparison alone may not reveal.
For example, an added section might indicate that the original draft was incomplete. It might also reflect a requirement introduced after the draft was generated. A change in tone might express a company-wide preference or a choice specific to one audience.
Those differences matter. I would not want the system to turn every edit into a universal instruction.
To make the log useful for evaluation, I would record the type of change, its significance, and the reviewer's explanation where available. Content corrections, missing components, tone adjustments, and formatting changes should remain distinguishable. A factual correction deserves different attention from a preference about heading style.
I would also keep quality judgments separate from the amount of editing. A draft that needs few changes may be useful, but acceptance alone does not establish that its claims are correct. The original evidence and the agreed criteria still matter.
Closing the Loop Requires More Than Collecting Feedback
The weekly review creates an opportunity for improvement. The next step is to decide what should change.
Recurring findings can inform revised instructions, updated templates, better reference examples, or changes to how the system retrieves information. Those improvements then need to be available to future runs. In this approach, we are refining the system's configuration and context, rather than retraining the underlying model.
This is also where I distinguish the feedback log from Ng's reflection pattern. Reflection can improve an output during a particular task. The feedback log supports improvement across repeated uses of the workflow.
For changes that affect shared standards or consequential behavior, I would have the responsible person review the proposed update before it becomes part of the system. Making a log visible supports oversight, but oversight still requires someone to review it and act.
I would then test the updated workflow against representative cases, including cases separate from the examples used to develop the change. That helps distinguish a broader improvement from a fix tailored to the latest complaint. Evaluation should continue as the system changes, with new failures becoming additional test cases.
This is the part I would build in upfront, even in a modest form: a record of the initial output, a clear definition of success, a way to capture corrections, and a repeatable process for checking whether changes help.
The Same Principle Applies to Internal and External Tools
The presentation example is an internal workflow, but the underlying approach is equally relevant to a customer-facing system.
For an internal drafting assistant, useful feedback might come from the final document and the corrections a colleague makes. For an external support tool, I would look at whether the issue was resolved, whether a person had to intervene, and what that person corrected.
The evaluation criteria should reflect the consequences of failure, not simply whether the tool is labeled internal or external. An internal research summary can still influence an important decision.
The timing of review matters too. A weekly feedback review is useful for finding patterns and improving the workflow. It should not replace checks or approvals needed before an output reaches a user or an action is taken.
I would decide those controls alongside the workflow itself: what can proceed automatically, what needs review, and what evidence the reviewer needs to make that decision.
Configure Existing Tools Before Building New Ones
There is another consideration: the capabilities available to teams keep changing.
When I first started this work, getting a general-purpose AI chat to follow company standards, tone, and output expectations required substantial effort. In my own work, that configuration has become easier over time.
Where an organization permits the relevant use, I often advise teams to thoroughly configure an existing environment before committing to a custom tool. ChatGPT and Claude both support project-specific instructions and reference material, giving teams ways to provide reusable context for their work.
I would start with clear instructions, approved examples, templates, and explicit quality criteria. Then I would test the configured environment against the actual task. A custom interface should solve a remaining problem, not compensate for configuration that has not been attempted.
The feedback mechanism still needs deliberate attention. I would verify how the chosen setup captures final edits, preserves useful records, and makes approved feedback available for subsequent work. I would not assume those connections exist simply because the chat has access to reference files.
A substantial part of my work involves sensitive information and environments with tighter restrictions. In those situations, the starting point is what the team is approved to use, which information the system may access, and what integrations and controls are permitted.
That may lead to an approved enterprise product, a custom internal implementation, or a combination of the two. Custom development still needs the appropriate review and approval. The same restrictions also need to cover feedback logs and corrected outputs, because those records may contain sensitive information.
Build for Improvement From the Beginning
What makes Andrew Ng's framework such a valuable reference is that it gives us a practical way to reason about AI systems. Reflection, tool use, planning, and collaboration describe how the work can be organized. Evaluation and error analysis help us decide how that organization should change.
My own experience has reinforced the importance of making those ideas part of the workflow from the beginning. Capture what the system produces. Understand what people change. Make the relevant steps inspectable. Turn recurring problems into specific improvements, then test whether those improvements hold up.
For internal tools and external products alike, I want more than an impressive first demonstration. I want a system whose performance we can understand and whose improvement we can verify.