Durable AI workflows: state, approvals and recovery

A vendor-onboarding workflow prepares a customer-record update, waits for a manager and sends the approved change to an external service. The service commits the update, but its response is lost. The worker crashes before recording success. When it restarts, should it send the update again? A saved conversation cannot answer whether the external effect already happened.

Model business state explicitly#

Persist business progress as structured state rather than inferring it from a transcript. A useful flow is RECEIVED → VALIDATED → PROPOSED → AWAITING_APPROVAL → READY_TO_COMMIT → SUBMITTED → CONFIRMED, with explicit branches for FAILED, CANCELLED and NEEDS_RECONCILIATION. Define which actor may make each transition and the evidence required. The model can propose a next step without owning the state machine.

LangGraph persistence stores thread state through checkpointers and separates cross-thread stores. An in-memory saver is useful for examples but is lost on restart; use a durable backing store for recovery. Thread IDs locate state, not authorize access to it. Protect checkpoints and traces with the same tenant and retention rules as the source data they contain.

Temporal documents durable workflow history and deterministic workflow requirements. Put nondeterministic model and network work behind the appropriate activity boundary and retain its result. Pin the chosen SDK behavior and plan compatibility for already-running workflows. A deployment that changes code can otherwise strand work that has been waiting for approval for days.

Trace the failure between external success and local knowledge#

The safe recovery depends on the receiving service. If it supports neither reliable idempotency nor lookup by a stable business reference, automatic retry can duplicate an effect. Put the item into reconciliation and make the residual risk visible. “Timeout” means the caller did not obtain a response; it does not mean the service performed no work.

Give one intended action one durable identity#

Generate and persist the business operation key before dispatch. Reuse that key and the same payload for retries of that intent. A new key on every retry defeats duplicate suppression. Scope uniqueness by tenant and operation where appropriate, and reject the same key with a different payload. The receiver must implement the check atomically; a client-side lookup followed by an unprotected write can race.

Stripe’s idempotent-request documentation is a concrete example of a server-side contract, including parameter consistency and bounded retention. Do not assume other APIs behave identically or retain keys forever. If recovery occurs after the receiver’s key retention window, use a durable operation reference or reconciliation before resubmitting.

Classify errors before retrying. A transient unavailable service may permit bounded backoff with jitter. An invalid payload or permission denial requires correction or review. An ambiguous result after dispatch requires checking what happened. Set attempt and elapsed-time limits, and retain the last known state so an exhausted retry policy produces an actionable queue item instead of lost work.

Resolve local dual writes without promising global exactly-once execution#

If local state and an outgoing event must agree, write both the business update and an outbox row in one database transaction. A dispatcher sends committed rows later. AWS’s transactional-outbox guidance explains this pattern and the need for idempotent consumers because delivery can still repeat. It does not make a third-party API part of that local transaction.

Use state versions or locks to prevent competing workers from dispatching incompatible proposals. A claimed job needs a lease or recovery policy so a dead worker does not hold it forever. A lease expiring does not prove the previous worker stopped; downstream idempotency remains necessary when two workers briefly overlap.

Resume the approved action, not a regenerated one#

An approval should identify the exact operation, recipient, fields, values, source versions, approver and expiry. Store the proposal rather than asking the model to recreate it after a restart. A new model response may choose a different recipient or amount even with similar instructions. Any material change should invalidate the old approval and follow the required review path.

After a long pause, recheck the approver’s authority, the initiating user’s scope, relevant record versions and current policy. A boolean resume signal is not sufficient transaction authorization. If a supplier is suspended while the workflow waits, the old approval should not silently override that new business fact.

LangGraph’s interrupt documentation notes that the node containing an interrupt restarts from its beginning on resume. Code before the pause can therefore run again. Keep non-idempotent effects out of replayed regions or protect them with a durable operation contract. A checkpoint is a recovery mechanism, not proof that every line before it executes only once.

Distinguish stopping from undoing#

Cancellation should stop new work, signal active workers and invalidate pending proposals. It may arrive after an external effect has committed. Inspect actual state before declaring the workflow cancelled without consequence. A compensation, such as reversing a draft record, is a new authorized business operation with its own failure modes; it is not a rewind button.

Some effects cannot be undone: an email was delivered or a downstream team acted on a message. Preserve the receipt, record the cancellation timing and assign reconciliation ownership. UI wording should distinguish “future steps stopped,” “change reversed” and “outcome still being checked.” This prevents operators from assuming that a closed task means no side effect occurred.

Test the exact crash windows#

Temporal’s activity documentation explains why activities should be idempotent: work may have occurred without its completion being recorded. Exercise that condition directly in a test environment. Also test expired idempotency keys, changed retry payloads, clock-sensitive approval expiry, checkpoint-store unavailability and backup restoration. A normal successful run does not test recovery.

Make unresolved outcomes visible#

Monitor the age of waiting approvals, submitted operations without receipts, retry exhaustion, duplicate suppression and reconciliation backlog. Define service expectations and an owner for every unresolved state. Keep enough redacted evidence to identify the external operation without turning telemetry into an uncontrolled copy of customer records.

Measure recovery time, lost work, duplicate effects and audit completeness under the tested scenarios. Report the actual boundaries and receiver contracts, not a blanket exactly-once guarantee. Reliability comes from a chain of explicit states, durable intent, safe retries and verified outcomes; the model transcript is only one piece of that chain.