Compound AI Systems: Add Components for a Demonstrated Need

Reviewed and updated October 8, 2026.

A compound AI system combines models with retrieval, tools, application logic or other models. The reason to add a component should be specific: obtain a current fact, enforce a rule, perform a permitted action or correct a demonstrated failure. A longer pipeline is not evidence of a better product.

Start with a simple application and a task-specific evaluation. Anthropic's guidance on effective agents recommends the simplest solution that works and describes the latency and cost tradeoffs of added complexity. The relevant comparison is useful work completed with acceptable failures and operating effort.

Give each component a contract

A retriever returns source identifiers and versions. A business API returns authorized records or an explicit error. A generator drafts an explanation. A validator checks defined constraints. The application owns access checks, accepted state changes and user-visible status.

For every boundary, define input schema, output schema, timeout, retry policy and failure behavior. Separate a tool's transport success from the business action's outcome. Record a trace identifier across components so an operator can find where a request stopped. Avoid retaining raw sensitive content merely because tracing is convenient.

An illustrative service-desk workflow

Suppose a service-desk assistant needs to draft a replacement request. This example is a proposed design, not a tested system. Begin with a model drafting from an approved procedure. If that draft meets the task, stop there.

If eligibility depends on the current asset record, add one read-only asset lookup. The API determines the user's permitted asset access. Supply the relevant record and procedure to the generator. Validate the draft's required fields, then show it for review. Add a write integration only if submitting the request is part of the agreed work and its authorization and duplicate handling are defined.

Compare versions by removing one component at a time. Does the asset lookup prevent a documented class of incorrect draft? Does another model checking the draft catch errors beyond deterministic field validation? Record the extra waiting time, inference use, integration failures and maintenance responsibility. A second model can repeat the first model's mistake; it is not an independent guarantee.

Choose a workflow before an open-ended agent

Fixed steps are appropriate when the process is known. A model can fill a bounded part of that workflow without selecting every action. An agent may be useful when the next step depends on findings that cannot be enumerated practically, but it still needs tools, stopping rules and action boundaries.

Multi-step state must survive failures when external actions are involved. Temporal's Activity documentation illustrates why retried external operations need idempotency. Durable orchestration does not by itself make a receiving API reject duplicate actions.

Account for the whole system

List owners for the model, data sources, integrations and runbook. Define a reduced mode if a dependency fails: a source link, a saved draft, a queued review or an unavailable response. Test that mode and ensure it cannot widen permissions. Keep a simpler version available for comparison and rollback.

See custom AI development for applications that need these boundaries. Request a project review with the workflow, current baseline and the failure a new component would address.