Durable-Execution | Roy Gabriel

Durable Agents with Temporal: Retries, Idempotency, and Long-Running State

Sat, 06 Dec 2025 12:00:00 -0500

Why this matters

Agents are often framed as “reason + tools.”

In production, the actual problem is execution:

calls fail
networks flake
credentials expire
humans need to approve steps
tasks take hours/days
systems restart
you need a forensic trail of what happened

If your agent runtime is “one process with a loop,” you will eventually lose state and do the wrong side effect twice.

This is why workflow engines exist.

Temporal’s model - durable workflows with deterministic execution and event history - maps incredibly well to tool-using agents. Temporal explicitly requires workflow code to be deterministic and provides APIs for versioning long-running workflows. [1][2]

This article is a production pattern: use Temporal to make agents durable.

TL;DR

Represent an agent run as a Temporal Workflow.
Make tool calls Activities (retryable, timeout-bounded).
Put side-effecting tools behind:
idempotency keys
preview -> apply
durable “exactly-once” semantics (from the workflow’s perspective)
Use Temporal’s retry policies for Activities and explicit failure handling. [3]
Use event history and replay for forensics (Temporal events are first-class). [4]
Use workflow versioning for safe evolution of long-running agents. [2]

Why agents need durable execution
Mapping an agent to Temporal
Determinism and why it matters
Retries, timeouts, and idempotency
Human-in-the-loop as a first-class step
Replay, audit, and debugging
Versioning: evolving agents safely
A production checklist
References

Why agents need durable execution

A few failure modes you’ll recognize:

Partial side effects

agent creates a ticket
process dies before storing the ticket ID
agent retries and creates a duplicate

Long-running waits

“wait for PR approvals”
“wait for a CI pipeline”
“wait for a meeting to complete” If your agent can’t wait durably, it becomes a polling daemon.

Human approval

Some steps should not be automated:

“apply to prod”
“send email”
“delete resources” You need durable pause/resume with clean audit.

Mapping an agent to Temporal

Workflow = agent run

One agent run becomes a single Temporal Workflow Execution. Temporal workflows are designed for long-running, durable coordination. [5]

Inside the workflow you model steps:

interpret goal
choose tools
call tools
react to results
request approvals
finalize output

Activities = tool calls and external IO

All external calls should be Activities:

MCP tool calls
HTTP calls
DB writes
notifications

Why? Activities are where retries and timeouts belong. Temporal defines retry policies as configuration for how and when to retry failures. [3]

Signals = external events

Use signals for:

human approvals
“cancel”
updated user intent
out-of-band events (“incident resolved”)

Queries = introspection

Expose workflow state:

current step
last tool call
pending approvals
budget remaining

Determinism and why it matters

Temporal requires workflow code to be deterministic. [1] Determinism is what allows Temporal to replay history and rebuild state after worker crashes.

Practical consequence:

Don’t do IO in workflow code.
Don’t read the current time directly in workflow code (use Temporal APIs).
Don’t call random generators without deterministic control.
Keep workflow logic as “orchestration,” not execution.

If you violate determinism, you can hit non-deterministic errors on replay. Temporal’s docs and community discussions emphasize this constraint and the need for careful changes. [1][2]

Retries, timeouts, and idempotency

Retry policies (Activities)

Temporal retry policies control backoff and retry behavior for activity failures. [3]

Use them intentionally:

retries for transient failures (rate limits, timeouts)
limited retries for “probably broken” failures
exponential backoff with jitter (avoid thundering herd)

Timeouts are not optional

Set explicit timeouts:

ScheduleToStart
StartToClose
ScheduleToClose

Without timeouts, retries can run “forever” in practice.

Idempotency keys for side effects

Your workflow can be retried/replayed. Your Activity can be retried. Upstream systems can time out after performing the operation.

For side-effecting tools:

generate an idempotency key in the workflow
pass it into the tool Activity
store “operation result” in workflow state

When the Activity retries, it reuses the key so the upstream system deduplicates.

This is the difference between “retries” and “duplicates.”

Human-in-the-loop as a first-class step

For dangerous operations:

pause
ask for approval with the plan summary
resume when approved

Temporal workflows can wait for signals without holding threads like a traditional process would.

This is one of the cleanest ways to build:

“preview -> approve -> apply” without building a bunch of custom state machinery.

Replay, audit, and debugging

Temporal events are recorded as part of the workflow’s event history. [4]

This yields production superpowers:

reconstruct exactly what happened
understand why a step was taken
replay a run to test a bug fix
implement “reset” patterns (carefully)

For agents, this is the difference between:

“the model did something weird” and
“step 7 called tool X with args Y after tool Z returned response R”

Versioning: evolving agents safely

Agent logic will change. Prompts will change. Tool contracts will change.

If you have long-running agents, you need a strategy that doesn’t break in-flight executions.

Temporal provides workflow versioning mechanisms because determinism means you can’t simply change workflow logic without thought. [2]

Production approach:

keep existing executions on old code paths
route new executions to new paths
migrate intentionally

This prevents “deploy broke every running workflow.”

A production checklist

Architecture

Agent runs modeled as workflows; tool calls as activities.
External events modeled as signals; state exposed via queries.

Determinism

No IO in workflow code (only orchestration).
Workflow changes use versioning strategy. [2]

Reliability

Retry policies defined for Activities. [3]
Timeouts defined and bounded.
Idempotency keys used for side-effecting actions.

Governance

Human approval gates exist for dangerous operations.
Audit trails include plan summaries and results.

Operability

Event history used for debugging and incident analysis. [4]

References

[1] Temporal - Workflow Definition (determinism requirement): https://docs.temporal.io/workflow-definition [2] Temporal Go SDK - Versioning (evolving deterministic workflows safely): https://docs.temporal.io/develop/go/versioning [3] Temporal - Retry Policies (how and when retries happen): https://docs.temporal.io/encyclopedia/retry-policies [4] Temporal - Events reference (event history): https://docs.temporal.io/references/events [5] Temporal - Workflows overview: https://docs.temporal.io/workflows