Go | Roy Gabriel

Cruvero - AI Agent Ecosystem Platform

Thu, 12 Feb 2026 19:25:00 -0500

Summary

Cruvero is a production-grade AI agent orchestration platform I designed and built from the ground up in Go. It treats durability, observability, and operational control as infrastructure guarantees, not library afterthoughts.

Where frameworks like LangGraph bolt checkpointing onto a graph abstraction, Cruvero inverts the model: Temporal’s battle-tested workflow engine is the foundation, and the agent abstraction compiles down to it. The result is a platform where retry logic, failure recovery, human-in-the-loop approval, and multi-agent coordination aren’t library features; they’re infrastructure guarantees backed by the same technology that runs Uber’s and Stripe’s most critical workflows.

The system currently spans 90,000+ lines of Go and TypeScript, with a comprehensive React UI, Kubernetes deployment via Helm and ArgoCD, and an enterprise MCP gateway architecture designed to support 1,000+ concurrent agents across 150+ integrations.

The Problem

Every major agent framework optimizes for the same thing: time-to-demo. Spin up a LangGraph chain, wire a few tools, get a result in 30 seconds. Impressive on a slide. Catastrophic in production.

The failure modes are predictable. An agent workflow running for 40 minutes crashes mid-execution; state is gone. A tool call to an external API times out; the entire run fails with no recovery. A billing-sensitive agent hallucinates a $50,000 API call; no cost guardrails existed to stop it. An agent enters a reasoning loop, calling the same tool 15 times with near-identical arguments; nothing detects the degeneration.

These aren’t edge cases. They’re the baseline reality of running AI agents at enterprise scale. Cruvero was built to make them structurally impossible.

Architecture

Cruvero’s architecture is layered around a single principle: every agent action is a Temporal activity, and every workflow survives infrastructure failure by default.

Core Runtime: The agent loop follows a deterministic decide → act → observe → repeat state machine. Each cycle produces an immutable DecisionRecord with content-addressed hashes of the prompt, state, tool schemas, and model config. This gives you complete forensic capability: for any decision an agent made, you can see the exact inputs, replay the decision with a different model, or run counterfactual analysis (“what if it had chosen differently at step 4?”).

Durable Execution: Temporal manages all workflow state. Agent runs survive process crashes, worker restarts, and infrastructure failures transparently. Long-running workflows (minutes to hours) use continue-as-new with automatic state compaction. There is zero data loss on agent failure, guaranteed by Temporal’s event sourcing, not by application-level retry logic.

Multi-Agent Coordination: A first-class supervisor pattern supports seven coordination strategies: delegate, broadcast, debate, pipeline, map-reduce, voting, and saga with compensation. Agents communicate through signals, shared blackboard state, and pub/sub events. A supervisor can launch child agents, aggregate their results, and handle partial failures; all as durable Temporal workflows with full replay capability.

Graph DSL & Workflow Engine: A custom graph DSL compiles structured execution plans (steps, conditional routes, parallel branches, join semantics, subgraphs) into Temporal workflows. Join modes include all, any, N-of-M, and voting. The visual workflow builder (React Flow) provides bidirectional serialization between the visual canvas and the underlying graph definition.

Neuro-Inspired Intelligence

This is the feature set that no other agent framework implements. Drawing from neuroscience and cognitive architecture research, this layer introduces eight subsystems that fundamentally change how agents reason, learn, and self-correct.

Metacognitive Monitoring: Modeled on prefrontal cortex performance monitoring. The system tracks tool call hashes, observation hashes, progress deltas, confidence entropy, and goal-drift scores (via embedding cosine similarity against the original prompt). When it detects degradation, such as repetition loops, stalled progress, drifting goals, or collapsing confidence, it triggers graduated backpressure: forced reflection, model escalation (swap to a more capable model mid-run), context reset, mandatory strategy pivots, or human escalation. No more agents spinning their wheels for 200 steps.

Attention-Weighted Context Windows: Inspired by hippocampal memory replay. Instead of dumping context linearly into the prompt, a multi-factor salience scorer (relevance, recency, confidence, usage frequency) re-ranks all memory before assembly. A dynamic token budget allocator shifts allocation by task phase. Planning phases boost semantic/procedural memory, execution phases boost tool schemas, and review phases boost episodic memory. An interference detector flags contradictory facts explicitly in the prompt rather than letting the LLM silently pick one.

Temporal Reasoning: Deadline-aware execution with soft and hard deadlines, graduated pressure levels (relaxed through critical), automatic model switching under time pressure, and structured time context injection into every prompt.

Agent Immune System: Anomaly signature tracking with automatic tool quarantine. When a tool’s behavior degrades or produces anomalous outputs, the immune system hashes the failure pattern, tracks hit counts, and quarantines the tool after a configurable threshold. A vaccination CLI injects procedural memory to teach agents how to work around quarantined capabilities.

Compositional Tool Synthesis: Meta-tools that chain multiple tool calls into atomic pipelines with pre/postcondition contracts, typed argument mapping, and enforcement of non-retryable errors on contract violations.

Federated Trust & Delegation: Trust scoring for multi-agent delegation. Agents build trust through successful task completion; supervisors automatically select agents based on capability manifests and accumulated trust scores. Delegation chains provide full accountability tracking for post-mortem analysis.

Execution Provenance Graph: A tamper-evident DAG tracking every action, decision, and data dependency in an agent run. Supports ancestor/descendant queries, subgraph extraction, and run diffing to compare two executions and identify the exact point of divergence.

Enterprise Governance

Cruvero’s enterprise hardening philosophy is “tenant isolation is a property of the architecture, not a feature.” Every boundary is enforced at the infrastructure layer.

Multi-Tenancy & Namespace Isolation: Temporal namespaces, Postgres row-level security, and network policies enforce tenant boundaries. Per-tenant model selection, tool access control, and resource quotas are infrastructure-level guarantees that cannot be bypassed by application code.

Rate Limiting, Quotas & Cost Guardrails: Per-decision cost tracking (estimated and actual) with configurable policies: max cost per run, max cost per step, prefer-cheaper-model flags. Budget enforcement halts runs before they exceed limits. A model catalog with pricing metadata enables real-time cost optimization across providers.

Audit Logging & Compliance: Every tool call, LLM invocation, and state mutation is authenticated, authorized, and recorded in a tamper-evident audit trail. SOC 2-ready export formats. PII detection across five enforcement boundaries (audit, output, tool I/O, memory, events) with 12 PII types, unified secret detection, Shannon entropy analysis, HMAC-based stable tokenization, and a risk scoring engine.

Security Hardening: OWASP Top 10 mitigations, RBAC with four role levels (Viewer, Editor, Admin, Super Admin), OIDC authentication, CSRF protection, input sanitization, and CSP headers.

Tool Ecosystem & MCP Integration

Semantic Tool Discovery: A three-stage pipeline (keyword search → embedding similarity → quality-weighted reranking) selects tools dynamically rather than dumping all tool schemas into every prompt. Tool quality tracking quarantines degraded tools automatically.

MCP Protocol: 150+ Model Context Protocol integrations (Notion, GitHub, AWS, Azure, O365, ServiceNow, Slack, and more) with standardized tool interfaces. The current architecture uses stdio subprocesses; the enterprise target architecture introduces a gateway-mediated Streamable HTTP model with per-integration scaling, Dragonfly response caching, circuit breakers, Vault-backed credential isolation, and KEDA autoscaling, designed for 1,000+ concurrent agents.

Event-Driven Architecture: NATS provides async event fan-out alongside Temporal’s durable execution. MCP server lifecycle management, embedding pipeline intake, audit/telemetry buffering, and external consumer subscriptions (Teams/Telegram bots, dashboards, webhook relays) all flow through NATS, without ever entering the workflow deterministic path.

Observability & Operations

Distributed Tracing: OpenTelemetry spans per decision cycle, tool call, memory operation, and MCP invocation. Full correlation IDs from workflow entry through every activity.

Structured Logging: Zap-based structured logging with per-tenant, per-run, and per-step context propagation.

Production API: RESTful API with automatic OpenAPI 3.1 documentation, SSE streaming for live run updates, and comprehensive endpoints for run management, approval workflows, replay, tracing, cost queries, and tool management.

React Operational UI: A full-featured React 18 / TypeScript interface replacing the original htmx console. Surfaces every runtime capability: run management with live SSE streaming, approval queues, replay console with counterfactual analysis, causal trace explorer, tool registry browser, memory explorer with salience scores, cost dashboards (ECharts), supervisor multi-agent visualization, visual workflow builder (React Flow), live workflow inspection, speculative execution, and differential model testing.

Kubernetes Deployment: Helm chart with environment-aware value overlays, ArgoCD ApplicationSet for GitOps promotion (dev/staging/prod), ServiceMonitor templates, and ingress configuration.

Key Decisions

Go over Python: Single-binary deploys, predictable latency, deterministic resource usage, and a strong concurrency model for managing hundreds of concurrent agent sessions. No GIL, no dependency hell, no runtime surprises.

Temporal over custom durability: Rather than implementing checkpointing, retry logic, and state recovery as library features, Cruvero delegates all of it to Temporal’s battle-tested workflow engine. This is the same infrastructure that runs mission-critical systems at companies processing millions of transactions per day.

Neuroscience-grounded intelligence: The cognitive architecture isn’t marketing. Each subsystem maps to a specific neuroscience principle (prefrontal monitoring, hippocampal salience, temporal reasoning, immune response). The result is agents that self-correct, learn from failures, and degrade gracefully, capabilities no other framework offers.

Context management as a competitive advantage: Most frameworks dump everything into the context window and pray. Cruvero’s context pipeline includes phase-aware budget allocation, five-component salience scoring, semantic tool search, interference detection, observation masking, and proactive compression triggers. The competitive analysis shows clear advantages over LangChain/LangGraph across every dimension.

Outcome

Cruvero runs production agent workloads with infrastructure-grade reliability guarantees. The platform handles long-running workflows (minutes to hours), survives arbitrary infrastructure failures without data loss, enforces per-tenant cost and security policies, and provides complete observability from workflow entry through every LLM decision and tool call.

The codebase represents 90,000+ lines of production code, 80%+ test coverage, comprehensive documentation published via Hugo, and a development methodology designed for systematic LLM-assisted engineering at scale.

Stack

Go · Temporal · PostgreSQL · NATS · React 18 · TypeScript · Vite · React Flow · ECharts · Tailwind CSS · Kubernetes · Helm · ArgoCD · Qdrant · Dragonfly · Ollama · OpenTelemetry · Zap · Keycloak · Docker

Go vs Spring Boot for Enterprise APIs: Cost, Performance, and Cloud-Native Ops

Sun, 01 Feb 2026 10:00:00 -0500

As-of note: This is a production engineering perspective, not a benchmark scoreboard. If you care about cost or p99 latency, measure your service with your dependencies and your deployment constraints.

Why this comparison keeps showing up

If you build enterprise APIs long enough, you’ll see the same pattern:

The “language choice” isn’t what breaks production.
The runtime envelope and operational model usually are.

When teams compare Go and Java Spring Boot, they’re often asking a more specific question:

“What will it cost to run this API at scale, and how predictable is it under real production conditions?”

Spring Boot’s value proposition is speed-to-service: stand-alone, production-grade Spring applications you can “just run,” with strong ecosystem defaults and integration breadth. [1]

Go’s value proposition is operational simplicity: compile to an executable, ship a small container, run with fewer moving pieces, and keep latency and resource usage easier to reason about. go build compiles packages into an executable. [5]

This article is about the production-relevant tradeoffs: cost/resource usage, performance under load, cloud-native deployability, and the “you will be on call for this” realities.

On code quality: This isn’t “Go good / Java bad.” It’s an observation about failure modes: framework-heavy stacks can hide complexity until it shows up in startup time, memory, and surprises under load. Go’s bias toward explicitness often makes problems easier to see and cheaper to operate, even before the codebase is perfect.

TL;DR

If your org is already Spring-heavy, Spring Boot can be the fastest path to a robust API, especially when you need Spring’s ecosystem (security, data, integrations). [1]
If you run many small services, care about density, or need fast scale-to-zero/scale-from-zero behavior, Go often has an operational edge due to simpler packaging and typically lower baseline resource footprint.
Kubernetes costs are strongly influenced by requests/limits and scheduling density, so baseline memory is often a bigger lever than micro-optimizing CPU. [7][8]
Both ecosystems support hardened container builds (including distroless) to reduce attack surface. [9][10]
Observability is excellent in both; Java has very mature zero-code instrumentation via the OpenTelemetry Java agent. [13][14] Go has strong SDK support and growing options for auto-instrumentation. [11]
“Best” depends on your constraints. The best move is to benchmark your service envelope and compare p95/p99 latency, RSS, startup, and error rates under load.

The cost model: what you actually pay for

In cloud and Kubernetes environments, cost is strongly driven by:

How many replicas you need
How much CPU/memory you request per replica
How quickly you can scale (up and down)
How much time you spend operating the service

Kubernetes scheduling and resource guarantees are based on requests and limits. Requests influence where Pods can be scheduled; limits cap what they can consume. [7][8]

That means your “baseline footprint” matters:

A service that requests 512Mi RAM even when idle reduces node density.
A service that requests 128Mi RAM allows more Pods per node.

A simple (illustrative) density example

Assume you run 100 replicas of an API, and memory is your limiting resource:

Case A: 100 × 512Mi = 51,200Mi ≈ 50Gi reserved
Case B: 100 × 128Mi = 12,800Mi ≈ 12.5Gi reserved

That’s a ~37.5Gi delta in reserved memory before you count overhead (sidecars, DaemonSets, kube-system). This is not “Go vs Java math.” It’s “baseline footprint sets cluster size.”

The point: cost discussions are often memory-and-startup discussions wearing a language-comparison mask.

Go’s production advantages (when they matter)

1) Packaging simplicity and deployment surface

Go’s toolchain compiles code into an executable (go build). [5] Go’s modern toolchain approach (including toolchain selection starting in recent Go releases) helps keep builds reproducible across environments. [6]

In practice, Go services often ship as:

a single process
a single container layer containing a single binary
minimal runtime dependencies

That tends to reduce:

container image complexity
“works on my machine” drift
runtime patch surface area

This matters most when you operate many services and want upgrades to be boring.

2) Fast start and “scale events”

In real systems, performance isn’t only request/response speed, it’s also how the service behaves during:

deployments
autoscaling
node drains
crashes

Go services commonly start quickly because they don’t require JVM warmup/classloading/JIT compilation. (Exact numbers vary; measure your service.)

Spring Boot can start fast enough for most use cases, but cold starts can become a visible factor when:

you scale from zero frequently (serverless-like patterns)
you do aggressive HPA scaling
you run lots of short-lived jobs

Spring Boot also supports building native images with GraalVM, which can materially improve startup and memory in some cases, but introduces different tradeoffs (build time, reflection limits, operational differences). [3][4]

3) Resource envelope predictability

For many “API gateway / orchestration / integration” services, CPU isn’t the bottleneck. Latency, network, and downstream behavior are.

Go’s strengths here tend to be:

predictable concurrency behavior
straightforward backpressure patterns (bounded queues, semaphores)
fewer runtime tuning knobs compared to JVM-heavy stacks

This is not “Go always uses less RAM.” It’s “Go often gives you a tighter baseline envelope for simpler services, which improves scheduling density.”

4) Cloud-native ergonomics: minimalism wins over time

Enterprise services accrete complexity over years. The less your runtime depends on:

classpath complexity
reflection-driven magic
extensive framework graphs

…the easier it is to keep production surprises rare.

Go’s bias toward explicit wiring tends to help with long-term operability, especially in platform/API layers where consistency matters.

Where Spring Boot is still the right tool

Spring Boot exists for a reason, and in many enterprises it’s still the correct default:

1) Ecosystem and “starter” leverage

Spring Boot’s opinionated defaults and starter ecosystem are an enormous accelerator for:

auth (OAuth2/OIDC)
data access and ORM patterns
enterprise integrations
standardized configuration and profiles

Spring Boot is explicitly designed to minimize configuration and help you ship “production-grade” applications quickly. [1]

If you already have:

shared Spring libraries
internal Spring starters
company-wide Spring conventions

…then choosing Go for “purity” can be expensive in human terms.

2) JVM performance can be excellent

For long-lived services under sustained load, HotSpot JIT compilation can deliver extremely strong performance, sometimes outperforming Go in CPU-bound or allocation-sensitive scenarios.

It’s a mistake to assume “compiled native binary” automatically means “faster.” The real question is: p99 latency, throughput per core, and behavior under GC pressure for your workload.

3) Operational maturity and tooling

Spring Boot has well-worn operational patterns:

actuator endpoints
consistent configuration patterns
deep tracing/profiling options
broad community knowledge

Also: if your org has deep Java on-call expertise, “operational simplicity” may already be solved socially.

Cloud-native reality: images, CVEs, and deploy surface

Distroless is not a Go-only advantage

A common Go pattern is “static binary + scratch/distroless.” But distroless images exist for Java too.

Distroless images contain only the application and its runtime dependencies, with no package manager and no shell, reducing attack surface. [9] The distroless project includes Java images as well. [10]

Operational implication: smaller, simpler images usually mean:

faster pulls and rollouts
fewer things to patch
fewer “shell inside container” habits (a feature, not a bug)

Whether you ship Go or Spring Boot, you can adopt hardened bases.

Two Dockerfile patterns (illustrative)

Go (multi-stage + distroless):

FROM golang:1.22-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags "-s -w" -o /out/api ./cmd/api

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/api /api
USER nonroot:nonroot
ENTRYPOINT ["/api"]

Spring Boot (JAR + distroless Java):

FROM eclipse-temurin:21-jdk AS build
WORKDIR /src
COPY . .
RUN ./mvnw -DskipTests package

FROM gcr.io/distroless/java21-debian12:nonroot
COPY --from=build /src/target/app.jar /app.jar
USER nonroot:nonroot
ENTRYPOINT ["java","-jar","/app.jar"]

The important part isn’t the exact base image, it’s the principle: reduce image surface area and keep the deploy artifact boring.

Observability and operations

Both ecosystems are strong here, but they differ in “how quickly can I get real telemetry.”

OpenTelemetry support

OpenTelemetry is the vendor-neutral standard for traces/metrics/logs. [11]

Go language docs: SDK + instrumentation guidance. [11]
Java language docs: SDK + instrumentation guidance. [12]

Java’s advantage: zero-code instrumentation

The OpenTelemetry Java agent can attach to Java applications and automatically instrument popular libraries via bytecode injection. [13] The OpenTelemetry Java instrumentation project provides the agent and broad library coverage. [14]

Practical implication: you can often get useful traces without touching code. That’s a meaningful ops advantage in large enterprises.

Go’s reality: explicit instrumentation (plus growing options)

Go’s OpenTelemetry SDK support is strong. [11] Go auto-instrumentation options exist and are improving, but your fastest path today is still typically:

instrument key inbound/outbound edges in code
standardize middleware across services
treat telemetry as part of the API contract

That’s not bad. It’s just a different default.

A decision matrix

Use this as a starting point, not a rule.

Constraint / Goal	Go tends to win	Spring Boot tends to win
Many small services, high density	✅ smaller baseline envelopes often help	⚠️ can be heavier per-service
Fast scale-from-zero, frequent redeploys	✅ typically quick startup	✅ with care; ✅✅ with native image tradeoffs [3][4]
Enterprise integration breadth	⚠️ you build more glue yourself	✅ Spring ecosystem leverage [1]
Team expertise	✅ if Go is your platform standard	✅ if Java/Spring is your standard
“Boring deployments”	✅ single binary patterns	✅ well-trodden JVM patterns
Zero-code observability	⚠️ emerging	✅ OTel Java agent maturity [13][14]
Long-lived CPU-heavy services	✅ sometimes	✅ JVM can be extremely strong

How to validate with a real experiment

If you want a decision you can defend, run a 2-4 hour experiment:

1) Define a representative endpoint mix

1 simple “health/read” endpoint
1 endpoint that hits your DB
1 endpoint that calls a downstream HTTP service
1 endpoint with payload validation + auth

2) Measure the four numbers that matter

Startup time (cold start to ready)
Steady-state RSS at idle
p95 / p99 latency under load
Error rate under load + partial downstream failure

3) Run the same load and failure profile

Use the same:

container runtime
resource requests/limits
ingress configuration
downstream simulators

4) Compare operational work, not only performance

How painful is debugging?
How much config is required?
How quickly can your team ship fixes safely?

This is where enterprise reality lives.

Common failure modes

Go pitfalls

Teams reinvent frameworks inconsistently across services.
Too much “just a handler” code without shared middleware for auth, limits, tracing, and error handling.
Ignoring backpressure (unbounded goroutines) → memory blowups.

Spring Boot pitfalls

Default dependency graphs grow quietly until startup time and memory become a problem.
Classpath/auto-config complexity makes “why did it do that?” debugging expensive.
Container runtime tuning gets deferred, then becomes urgent during cost reviews.

Both ecosystems

No explicit timeouts (inbound and outbound).
No limits or budgets.
No telemetry until after the first incident.

Closing thought

If your enterprise APIs are:

small, numerous, latency-sensitive, and cost-sensitive
…Go is often a strong default.

If your enterprise APIs are:

integration-heavy, domain-rich, and built on existing Spring conventions
…Spring Boot is usually the shortest path to “production-grade.”

The best answer is the one you can operate confidently, on call, at scale.

References

Spring Boot project overview:
Spring Boot reference: Graceful Shutdown:
Spring Boot reference: GraalVM Native Images:
GraalVM guide: Build a Spring Boot app into a native executable:
Go tutorial: Compile and install the application (go build produces an executable):
Go docs: Toolchains and the go command:
Kubernetes docs: Resource Management for Pods and Containers (requests/limits):
Google Cloud: Kubernetes best practices for resource requests and limits:
Distroless container images (project overview):
Distroless Java images:
OpenTelemetry Go docs:
OpenTelemetry Java docs:
OpenTelemetry Java Agent (zero-code):
OpenTelemetry Java instrumentation (agent JAR + library coverage):

MCP Servers in Production: Hardening, Backpressure, and Observability (Go)

Sat, 31 Jan 2026 09:00:00 -0500

As-of note: MCP is evolving. This article references the MCP specification versioned 2025-11-25 and related docs; verify details against the current spec before shipping changes. [1][2][4]

Why this matters

Most “agent demos” fail in production for boring reasons: missing timeouts, unbounded concurrency, ambiguous tool interfaces, and logging that accidentally turns into data exfiltration.

An MCP server isn’t “just an integration.” It’s a capability boundary between an LLM host (IDE, desktop app, agent runner) and the real world: files, APIs, databases, tickets, home automation, and anything else you wire up. MCP uses JSON-RPC 2.0 messages over transports like stdio (local) and Streamable HTTP (remote). [1][2][5]

That means an MCP server is:

an API gateway for tools
a policy enforcement point (whether you intended it or not)
a reliability hotspot (tool calls are where latency and failure concentrate)
a security hotspot (tools are where “read” becomes “exfil” and “write” becomes “impact”)

This post is a pragmatic checklist + a set of Go patterns to harden an MCP server so it keeps working when it’s under real load, and remains safe when the model gets “creative.”

TL;DR

Treat tool inputs as untrusted. Validate and constrain everything.
Put budgets everywhere: timeouts, concurrency limits, rate limits, and payload caps.
Build for partial failure: retries, idempotency keys, circuit breaking, fallbacks.
Log like a security engineer: structured, redacted, auditable, and useful. [11]
Instrument with traces/metrics early; “we’ll add telemetry later” is a trap. [13]
Prefer Go for MCP servers because deployment and operational behavior are predictable: single binary, fast startup, structured concurrency via context, and a strong standard library.

A production mental model for MCP servers
Threat model: what actually goes wrong
Hardening layer 1: identity and authorization
Hardening layer 2: tool contracts that resist ambiguity
Hardening layer 3: budgets and backpressure
Hardening layer 4: safe networking and SSRF containment
Hardening layer 5: observability without leaking secrets
Hardening layer 6: versioning and rollout discipline
A production checklist
References

A production mental model for MCP servers

MCP’s docs describe a host (the AI application), a client (connector inside the host), and servers (capabilities/providers). Servers can be “local” (stdio) or “remote” (Streamable HTTP). [2][3]

Here’s the production mental model that matters:

Your MCP server is a tool gateway.
Every tool is effectively an RPC method exposed to an agent. MCP uses JSON-RPC 2.0 semantics for requests/responses/notifications. [1][5]
LLM tool arguments are not trustworthy.
Even if the LLM is “helpful,” arguments can be malformed, overbroad, or dangerous, especially under prompt injection or user-provided hostile input.
The host UI is not a security boundary.
The spec emphasizes user consent and tool safety, but the protocol can’t enforce your policy for you. You still need server-side controls. [1]
Transport changes your blast radius, not your responsibilities.
Stdio reduces network exposure, but doesn’t remove safety requirements. Streamable HTTP adds multi-client/multi-tenant concerns and requires real auth. [2][3]

If you remember nothing else: treat the MCP server like a production API you’d be willing to put on call for.

Threat model: what actually goes wrong

When MCP servers cause incidents, it’s usually one of these:

1) Input ambiguity → destructive actions

A “delete” tool with optional filters
A “run command” tool with free-form strings
A “sync” tool that can touch thousands of objects

Mitigation: schema + semantic validation, safe defaults, two-phase commit patterns (preview then apply), and explicit “danger gates.”

2) Prompt injection → tool misuse

The model can be tricked into calling tools with attacker-provided arguments. If your tool can read internal data or call internal APIs, you’ve created an exfil path.

Mitigation: least privilege, allowlists, strong auth, egress controls, and redaction.

3) SSRF / network pivoting

Any tool that fetches URLs, loads webhooks, or calls dynamic endpoints can be abused to hit internal networks or metadata endpoints. OWASP treats SSRF as a major category for a reason. [10]

Mitigation: deny-by-default networking (CIDR blocks, DNS/IP resolution checks, allowlisted destinations).

4) Unbounded concurrency → resource collapse

Agents can fire tools in parallel. Without limits you’ll blow up:

API quotas
DB connections
CPU/memory
downstream latency

Mitigation: per-tenant rate limiting, concurrency caps, queues, and backpressure.

5) “Helpful logs” → data leak

Tool arguments and tool responses often contain secrets, tokens, or private data. If you log everything, you’ve built an involuntary data lake.

Mitigation: structured + redacted logging, security logging guidelines, and minimal retention. [11][12]

Hardening layer 1: identity and authorization

If you run Streamable HTTP, assume:

multiple clients
untrusted networks
tokens will leak eventually

MCP’s architecture guidance recommends standard HTTP authentication methods and mentions OAuth as a recommended way to obtain tokens for remote servers. [2][3]

Practical rules

Authenticate every request.
Use bearer tokens or mTLS depending on environment.
Authorize per tool.
“Authenticated” ≠ “allowed to run delete_everything”.
Prefer short-lived tokens and rotate them. [12]
Multi-tenant? Put the tenant identity into:
- auth token claims, or
- an explicit, validated tenant header (signed), then
- enforce it everywhere.

Go pattern: a minimal auth middleware skeleton (HTTP transport)

This is not a full MCP implementation, just the hardening pattern you’ll wrap around your MCP handler.

// Pseudocode-ish middleware skeleton. Replace verifyToken with your auth logic.
func authMiddleware(next http.Handler) http.Handler {
 return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 token := strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ")
 if token == "" {
 http.Error(w, "missing auth", http.StatusUnauthorized)
 return
 }

 ident, err := verifyToken(r.Context(), token) // includes tenant + scopes
 if err != nil {
 http.Error(w, "invalid auth", http.StatusUnauthorized)
 return
 }

 ctx := context.WithValue(r.Context(), ctxKeyIdentity{}, ident)
 next.ServeHTTP(w, r.WithContext(ctx))
 })
}

Key point: authorization should happen after you parse the requested tool name, but before you execute anything.

Hardening layer 2: tool contracts that resist ambiguity

Most MCP tool failures are self-inflicted: tool interfaces are too vague.

Design tools like production APIs

Bad tool signature:

run(command: string)

Better:

run_command(program: enum, args: string[], cwd: string, timeout_ms: int, dry_run: bool)

Why it’s better:

forces structure
allows you to enforce allowlists
gives you timeouts and safe defaults

Add a “preview → apply” flow for risky tools

For any tool that writes data or triggers side effects, do a two-step approach:

plan_* returns a machine-readable plan + a plan_id
apply_* requires plan_id and optional user confirmation token

This mirrors how we run infra changes (plan/apply) and dramatically reduces accidental blast radius.

Hardening layer 3: budgets and backpressure

Production systems are budget systems.

If you don’t set explicit budgets, your MCP server will eventually allocate them for you via outages.

Budget checklist

Server timeouts (header read, request read, write, idle)
Request body caps
Outbound timeouts to dependencies
Concurrency caps per tool and per tenant
Rate limits per tenant and per identity
Queue limits (bounded channels) to avoid memory blowups
Circuit breaking for flaky downstream dependencies

Go: server timeouts are not optional

Go’s net/http provides explicit server timeouts; leaving them at zero is a common footgun. [6][7]

srv := &http.Server{
 Addr: ":8080",
 Handler: handler, // your MCP handler + middleware
 ReadHeaderTimeout: 5 * time.Second,
 ReadTimeout: 30 * time.Second,
 WriteTimeout: 30 * time.Second,
 IdleTimeout: 60 * time.Second,
}
log.Fatal(srv.ListenAndServe())

Go: propagate cancellation everywhere with `context`

context.Context is the backbone of “structured concurrency” in Go: deadlines and cancellation signals flow through your call stack. [8][9]

Rule: every tool execution must accept a context.Context, and every outbound call must honor it.

func (s *Server) toolCall(ctx context.Context, req ToolRequest) (ToolResponse, error) {
 ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
 defer cancel()

 // ... outbound calls use ctx
 return s.integration.Do(ctx, req)
}

Go: per-tenant rate limiting with `x/time/rate`

golang.org/x/time/rate implements a token bucket limiter. [9]

type limiters struct {
 mu sync.Mutex
 m map[string]*rate.Limiter
}

func (l *limiters) get(key string) *rate.Limiter {
 l.mu.Lock()
 defer l.mu.Unlock()
 if l.m == nil { l.m = map[string]*rate.Limiter{} }
 if lim, ok := l.m[key]; ok { return lim }

 // Example: 5 req/sec with bursts up to 10
 lim := rate.NewLimiter(5, 10)
 l.m[key] = lim
 return lim
}

func rateLimitMiddleware(lims *limiters, next http.Handler) http.Handler {
 return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 ident := mustIdentity(r.Context())
 if !lims.get(ident.TenantID).Allow() {
 http.Error(w, "rate limited", http.StatusTooManyRequests)
 return
 }
 next.ServeHTTP(w, r)
 })
}

Backpressure: choose a policy

When you’re overloaded, you need a policy. Pick one explicitly:

Fail fast with 429 / “busy” (simplest, safest)
Queue with bounded depth (more complex; must cap memory)
Degrade by disabling expensive tools first

The “fail fast” approach is often correct for tool gateways.

Hardening layer 4: safe networking and SSRF containment

If any tool can fetch a user-provided URL or call a user-influenced endpoint, SSRF is on the table. [10]

SSRF containment strategies that actually work

OWASP’s SSRF guidance boils down to a few themes: don’t trust user-controlled URLs, use allowlists, and enforce network controls. [10]

In practice, for MCP servers:

Prefer allowlists over blocklists.
“Only these domains” beats “block internal IPs.” Attackers are creative.
Resolve and validate IPs before dialing.
DNS can be weaponized. Validate the final destination IP (and re-validate on redirects).
Disable redirects or re-validate each hop.
Redirect chains are SSRF’s favorite tool.
Enforce egress policy at the network layer too.
Kubernetes NetworkPolicies / firewall rules are your last line of defense.

Go pattern: an outbound HTTP client with strict timeouts

client := &http.Client{
 Timeout: 10 * time.Second, // whole request budget
 Transport: &http.Transport{
 Proxy: http.ProxyFromEnvironment,
 DialContext: (&net.Dialer{
 Timeout: 5 * time.Second,
 KeepAlive: 30 * time.Second,
 }).DialContext,
 TLSHandshakeTimeout: 5 * time.Second,
 ResponseHeaderTimeout: 5 * time.Second,
 ExpectContinueTimeout: 1 * time.Second,
 MaxIdleConns: 100,
 IdleConnTimeout: 90 * time.Second,
 },
}

Then wrap URL validation around any request creation. Keep it boring and strict.

Hardening layer 5: observability without leaking secrets

Telemetry is how you prove:

you’re within budgets
tools behave as expected
failures are localized
incidents can be diagnosed without “ssh and guess”

But logging is also where teams accidentally leak sensitive data.

OWASP’s logging guidance emphasizes logging that supports detection/response while avoiding sensitive data exposure. [11] Pair that with secrets management discipline. [12]

What to measure (minimum viable MCP telemetry)

Counters

tool_calls_total{tool, tenant, status}
auth_failures_total{reason}
rate_limited_total{tenant}

Histograms

tool_latency_seconds{tool}
outbound_latency_seconds{dependency}

Gauges

in_flight_tool_calls{tool}
queue_depth{tool}

Trace boundaries

Instrument:

request → tool routing
tool execution span
downstream calls span

OpenTelemetry’s Go docs show how to add instrumentation and emit traces/metrics. [13]

Logging rules that save you later

Use structured logging (JSON).
Add correlation IDs (trace IDs) to logs.
Redact:
- Authorization headers
- tokens
- cookies
- tool payload fields known to contain secrets
Log events, not raw payloads:
- “tool X called”
- “resource Y read”
- “write operation requested (dry_run=true)”

Audit logs

For high-impact tools, write an append-only audit record:
- who (identity)
- what (tool + parameters summary)
- when
- result (success/failure)
- plan_id / idempotency_key

Audit logs should be treated as security data.

Hardening layer 6: versioning and rollout discipline

MCP uses string-based version identifiers like YYYY-MM-DD to represent the last date of backwards-incompatible changes. [4]

That’s helpful, but it doesn’t solve the operational problem:

clients upgrade at different times
schema changes drift
hosts differ in which capabilities they support

Practical compatibility rules

Pin your server’s supported protocol version and expose it in health or diagnostics.
Add contract tests that run against:
- one “current” client
- one “previous” client version
Support additive changes first:
- new tools
- new optional fields
Use feature flags for risky tools.

Rollout like a platform team

Canaries for remote servers
“Shadow mode” for new tools (log what would happen)
Slow ramp with budget monitoring

A production checklist

If you’re building (or inheriting) an MCP server, run this checklist:

Safety

Tool contracts are structured (no free-form “do anything” strings).
Every tool has a safe default (dry_run=true, limit required, etc.).
Destructive tools require a plan/apply step (or explicit confirmation gates).
Tool inputs are validated and bounded (length, ranges, enums).

Identity & access

Remote transport requires authentication and per-tool authorization.
Tokens are short-lived and rotated; secrets are not in source control. [12]
Tenant identity is enforced at every access point (not “best effort”).

Budgets & resilience

HTTP server timeouts are configured. [6][7]
Outbound clients have timeouts and connection limits.
Rate limiting exists per tenant/identity. [9]
Concurrency caps exist per tool; overload behavior is explicit (fail fast / queue).
Retries are bounded and idempotent where side effects exist.

Networking

URL fetch tools have allowlists and SSRF protections. [10]
Redirect policies are explicit (disabled or re-validated).
Egress is constrained at the network layer (not only in code).

Observability

Metrics cover tool calls, latency, errors, and rate limiting.
Tracing exists across tool execution and downstream calls. [13]
Logs are structured, correlated, and redacted. [11]
Audit logging exists for high-impact tools.

Operations

Health checks and readiness checks exist.
Configuration is explicit and validated on startup.
Versioning strategy is documented and tested. [4]

References

Model Context Protocol (MCP) Specification (version 2025-11-25): https://modelcontextprotocol.io/specification/2025-11-25
MCP Architecture Overview (participants, transports, concepts): https://modelcontextprotocol.io/docs/learn/architecture
MCP Transport details (Streamable HTTP transport overview): https://modelcontextprotocol.io/specification/2025-03-26/basic/transports
MCP Versioning: https://modelcontextprotocol.io/specification/versioning
JSON-RPC 2.0 Specification: https://www.jsonrpc.org/specification
Go net/http package documentation: https://pkg.go.dev/net/http
Cloudflare: “The complete guide to Go net/http timeouts”: https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/
Go context package documentation: https://pkg.go.dev/context
Go x/time/rate documentation: https://pkg.go.dev/golang.org/x/time/rate
OWASP SSRF Prevention Cheat Sheet / SSRF category references:

OWASP Logging Cheat Sheet (security-focused logging guidance): https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
Secrets management guidance:

OWASP Secrets Management Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
Kubernetes “Good practices for Kubernetes Secrets”: https://kubernetes.io/docs/concepts/security/secrets-good-practices/

OpenTelemetry Go instrumentation docs: https://opentelemetry.io/docs/languages/go/instrumentation/

Agent Observability That Doesn't Lie

Sat, 20 Dec 2025 12:00:00 -0500

Why this matters

Most “agent observability” is either:

too shallow (a chat transcript and a couple logs), or
too noisy (every token logged, every tool payload stored, no signal)

Neither works in production.

If you’re serious about operating agents, you need observability that answers three questions quickly:

What happened? (forensics)
Why did it happen? (debuggability)
How often does it happen? (reliability)

OpenTelemetry exists to standardize how you instrument, generate, and export telemetry across traces, metrics, and logs. [1] W3C Trace Context defines how trace context propagates across service boundaries. [2]

Agents add two new requirements:

tool calls are part of your “distributed trace”
“decisioning” is a first-class component (not just business logic)

This article is a practical blueprint.

TL;DR

Instrument agents like distributed systems:
traces for causality (what triggered what)
metrics for health (p95 latency, error rates)
logs for human context (but redacted)
Propagate a single trace across:
agent runtime -> MCP gateway -> MCP tool servers -> upstream APIs
Capture decision summaries, not chain-of-thought.
Treat cost as a production signal: emit per-run and per-tool cost metrics.
Use semantic conventions where possible to keep telemetry queryable. [3]
Don’t turn observability into a data breach: OWASP highlights sensitive info disclosure and prompt injection as key risks. [7]

What to observe in an agent system
A trace model for agents
Metrics that matter
Logs and redaction
Audit events vs debug logs
Dashboards and alerts
A production checklist
References

What to observe in an agent system

Agents have four observable subsystems:

Planner/Reasoner (creates the plan, chooses tools)
Tool execution (calls MCP tools and interprets results)
Memory/state (what was stored or retrieved)
Policy/budget (what was allowed or blocked)

If you only observe #2, you’ll miss why the agent chose the wrong tool. If you only observe #1, you’ll miss production failures.

You need the full chain.

A trace model for agents

The core idea

A single “agent run” is a distributed trace:

it spans model calls
tool calls
downstream system calls

Use W3C Trace Context (traceparent, tracestate) to propagate the trace across boundaries. [2]

Suggested spans (minimum viable)

Root span

agent.run
attributes: agent.name, tenant, user, session, goal_hash

Planner

agent.plan
attributes: planner.model, plan.step_count

Model calls

llm.call
attributes: model, prompt_tokens, completion_tokens, latency_ms

Tool selection

agent.tool_select
attributes: selector.version, candidate_count, selected_count

Tool call

tool.call
attributes: tool.name, tool.class (read/write/danger), tool.server, status

Policy

policy.check
attributes: policy.rule_id, decision (allow/deny), reason_code

Memory

memory.read / memory.write
attributes: store, keys, bytes

Why spans > logs

Spans give you causality:

which tool call caused a failure
which step blew the budget
which upstream dependency was slow

With OpenTelemetry, you can emit traces and metrics using the same SDK approach. [1][4]

Metrics that matter

Tool health metrics

tool_calls_total{tool,status}
tool_latency_ms_bucket{tool}
tool_timeouts_total{tool}
tool_retries_total{tool}

Agent run health metrics

agent_runs_total{status}
agent_run_latency_ms_bucket{agent}
agent_steps_total_bucket{agent}

Cost metrics (treat cost like reliability)

llm_tokens_total{model,type=prompt|completion}
llm_cost_usd_total{model}
run_cost_usd_bucket{agent}

Policy metrics

policy_denied_total{rule_id}
danger_tool_attempt_total{tool}

Semantic conventions help your metrics stay queryable and consistent across systems. OpenTelemetry documents semantic conventions for HTTP spans/metrics, for example. [3][5]

Logs and redaction

Logs should add human context, not become a data lake of secrets.

Rules I like:

Do not log prompts by default.
Do not log tool payloads by default.
Log summaries and hashes:
goal_hash, plan_hash, tool_args_hash
Log structured error reasons:
validation_error, upstream_rate_limited, auth_failed, policy_denied

For agent systems, OWASP highlights sensitive information disclosure and insecure output handling. Logging is one of the easiest ways to accidentally create both. [7]

“Debug mode” that isn’t dangerous

If you must support deeper logs:

only enable per tenant/user for a limited window
auto-expire
redact aggressively
never store raw secrets

Audit events vs debug logs

Treat them as different products:

Audit events (for governance)

immutable-ish records of side effects
minimal sensitive data
always on
long retention

Example audit fields:

who: tenant/user/client
what: tool + action class (create/update/delete)
when: timestamp
where: environment
result: success/failure
resource IDs (safe identifiers)
idempotency keys / plan IDs

Debug logs (for engineers)

short retention
more context
highly controlled access

Mixing these two is how you end up with “SharePoint logs full of PII” and no one wants to touch them.

Dashboards and alerts

Dashboards (start simple)

Tool reliability

top tools by error rate
top tools by p95 latency
timeouts per tool

Agent success

success rate by agent type
“stuck runs” (runs exceeding max duration)
average steps per run

Cost

cost per run
cost per tenant
top drivers (which tools/model calls)

Alerts (avoid noise)

Alert on what is actionable:

tool error rate spikes for critical tools
tool latency p95 spikes beyond SLO
budget exceeded spike (runaway behavior)
policy denied spike (possible prompt injection attempt)

If you use SLOs and error budgets, Google’s SRE material is a practical reference for turning SLOs into alerting strategies. [6]

A production checklist

Tracing

Every agent run has a trace ID.
Trace context propagates across MCP boundaries (W3C Trace Context). [2]
Tool calls are spans with stable tool identifiers.

Metrics

Tool success/error/latency metrics exist.
Agent run success/latency/steps metrics exist.
Cost metrics exist and are monitored.

Logging

Default logs are redacted summaries, not raw payloads.
Debug logging is time-bounded and access-controlled.

Audit

Audit events exist for all side-effecting tools.
Audit records include “who/what/when/result” without leaking secrets.

Security

Observability does not become a secret exfil path (OWASP risks considered). [7]

References

[1] OpenTelemetry - Documentation (overview): https://opentelemetry.io/docs/ [2] W3C - Trace Context: https://www.w3.org/TR/trace-context/ [3] OpenTelemetry - Semantic conventions for HTTP (spans/metrics/logs): https://opentelemetry.io/docs/specs/semconv/http/ [4] OpenTelemetry Go - Instrumentation docs: https://opentelemetry.io/docs/languages/go/instrumentation/ [5] OpenTelemetry - Semantic conventions for HTTP metrics: https://opentelemetry.io/docs/specs/semconv/http/http-metrics/ [6] Google SRE Workbook - Alerting on SLOs: https://sre.google/workbook/alerting-on-slos/ [7] OWASP - Top 10 for Large Language Model Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/

Durable Agents with Temporal: Retries, Idempotency, and Long-Running State

Sat, 06 Dec 2025 12:00:00 -0500

Why this matters

Agents are often framed as “reason + tools.”

In production, the actual problem is execution:

calls fail
networks flake
credentials expire
humans need to approve steps
tasks take hours/days
systems restart
you need a forensic trail of what happened

If your agent runtime is “one process with a loop,” you will eventually lose state and do the wrong side effect twice.

This is why workflow engines exist.

Temporal’s model - durable workflows with deterministic execution and event history - maps incredibly well to tool-using agents. Temporal explicitly requires workflow code to be deterministic and provides APIs for versioning long-running workflows. [1][2]

This article is a production pattern: use Temporal to make agents durable.

TL;DR

Represent an agent run as a Temporal Workflow.
Make tool calls Activities (retryable, timeout-bounded).
Put side-effecting tools behind:
idempotency keys
preview -> apply
durable “exactly-once” semantics (from the workflow’s perspective)
Use Temporal’s retry policies for Activities and explicit failure handling. [3]
Use event history and replay for forensics (Temporal events are first-class). [4]
Use workflow versioning for safe evolution of long-running agents. [2]

Why agents need durable execution
Mapping an agent to Temporal
Determinism and why it matters
Retries, timeouts, and idempotency
Human-in-the-loop as a first-class step
Replay, audit, and debugging
Versioning: evolving agents safely
A production checklist
References

Why agents need durable execution

A few failure modes you’ll recognize:

Partial side effects

agent creates a ticket
process dies before storing the ticket ID
agent retries and creates a duplicate

Long-running waits

“wait for PR approvals”
“wait for a CI pipeline”
“wait for a meeting to complete” If your agent can’t wait durably, it becomes a polling daemon.

Human approval

Some steps should not be automated:

“apply to prod”
“send email”
“delete resources” You need durable pause/resume with clean audit.

Mapping an agent to Temporal

Workflow = agent run

One agent run becomes a single Temporal Workflow Execution. Temporal workflows are designed for long-running, durable coordination. [5]

Inside the workflow you model steps:

interpret goal
choose tools
call tools
react to results
request approvals
finalize output

Activities = tool calls and external IO

All external calls should be Activities:

MCP tool calls
HTTP calls
DB writes
notifications

Why? Activities are where retries and timeouts belong. Temporal defines retry policies as configuration for how and when to retry failures. [3]

Signals = external events

Use signals for:

human approvals
“cancel”
updated user intent
out-of-band events (“incident resolved”)

Queries = introspection

Expose workflow state:

current step
last tool call
pending approvals
budget remaining

Determinism and why it matters

Temporal requires workflow code to be deterministic. [1] Determinism is what allows Temporal to replay history and rebuild state after worker crashes.

Practical consequence:

Don’t do IO in workflow code.
Don’t read the current time directly in workflow code (use Temporal APIs).
Don’t call random generators without deterministic control.
Keep workflow logic as “orchestration,” not execution.

If you violate determinism, you can hit non-deterministic errors on replay. Temporal’s docs and community discussions emphasize this constraint and the need for careful changes. [1][2]

Retries, timeouts, and idempotency

Retry policies (Activities)

Temporal retry policies control backoff and retry behavior for activity failures. [3]

Use them intentionally:

retries for transient failures (rate limits, timeouts)
limited retries for “probably broken” failures
exponential backoff with jitter (avoid thundering herd)

Timeouts are not optional

Set explicit timeouts:

ScheduleToStart
StartToClose
ScheduleToClose

Without timeouts, retries can run “forever” in practice.

Idempotency keys for side effects

Your workflow can be retried/replayed. Your Activity can be retried. Upstream systems can time out after performing the operation.

For side-effecting tools:

generate an idempotency key in the workflow
pass it into the tool Activity
store “operation result” in workflow state

When the Activity retries, it reuses the key so the upstream system deduplicates.

This is the difference between “retries” and “duplicates.”

Human-in-the-loop as a first-class step

For dangerous operations:

pause
ask for approval with the plan summary
resume when approved

Temporal workflows can wait for signals without holding threads like a traditional process would.

This is one of the cleanest ways to build:

“preview -> approve -> apply” without building a bunch of custom state machinery.

Replay, audit, and debugging

Temporal events are recorded as part of the workflow’s event history. [4]

This yields production superpowers:

reconstruct exactly what happened
understand why a step was taken
replay a run to test a bug fix
implement “reset” patterns (carefully)

For agents, this is the difference between:

“the model did something weird” and
“step 7 called tool X with args Y after tool Z returned response R”

Versioning: evolving agents safely

Agent logic will change. Prompts will change. Tool contracts will change.

If you have long-running agents, you need a strategy that doesn’t break in-flight executions.

Temporal provides workflow versioning mechanisms because determinism means you can’t simply change workflow logic without thought. [2]

Production approach:

keep existing executions on old code paths
route new executions to new paths
migrate intentionally

This prevents “deploy broke every running workflow.”

A production checklist

Architecture

Agent runs modeled as workflows; tool calls as activities.
External events modeled as signals; state exposed via queries.

Determinism

No IO in workflow code (only orchestration).
Workflow changes use versioning strategy. [2]

Reliability

Retry policies defined for Activities. [3]
Timeouts defined and bounded.
Idempotency keys used for side-effecting actions.

Governance

Human approval gates exist for dangerous operations.
Audit trails include plan summaries and results.

Operability

Event history used for debugging and incident analysis. [4]

References

[1] Temporal - Workflow Definition (determinism requirement): https://docs.temporal.io/workflow-definition [2] Temporal Go SDK - Versioning (evolving deterministic workflows safely): https://docs.temporal.io/develop/go/versioning [3] Temporal - Retry Policies (how and when retries happen): https://docs.temporal.io/encyclopedia/retry-policies [4] Temporal - Events reference (event history): https://docs.temporal.io/references/events [5] Temporal - Workflows overview: https://docs.temporal.io/workflows

From Stdio to Enterprise: The MCP Gateway Pattern

Sat, 22 Nov 2025 12:00:00 -0500

As-of note: MCP evolves quickly. This article references the MCP spec revision 2025-11-25. Validate details against the current spec before shipping changes. [1][2][3]

Why this matters

Local MCP servers over stdio are an amazing developer experience: you install a tool server, the host (Claude Desktop / Claude Code / an agent runtime) launches it, and you’re productive in minutes. [2]

But as soon as MCP becomes shared infrastructure - multiple clients, multiple users, multiple environments - the “local tool server” model runs into the same constraints every integration layer hits:

Who is allowed to call what tool?
How do you prevent one noisy user from melting shared dependencies?
How do you audit tool side effects?
How do you roll out tool changes without breaking clients?
How do you keep secrets out of prompts, logs, and screenshots?

This is where the MCP Gateway Pattern shows up.

A gateway is not “another service.” It’s a capability boundary: the place where you enforce policy, budgets, and observability for tool use at scale.

TL;DR

Stdio is great for local, single-user, low-blast-radius setups.
HTTP transports (Streamable HTTP) enable multi-client servers - but they also require real auth and multi-tenant safety. [2][3]
An MCP gateway sits between clients and tool servers to provide:
authentication & authorization
tenant isolation
rate limits / concurrency / cost budgets
consistent tool schemas + safety gates
audit logs and observability
routing, versioning, rollout controls
Build the gateway to be boring: small surface area, strict validation, explicit policies, great telemetry.

When stdio stops being enough
The MCP Gateway Pattern
Responsibilities of a gateway
Reference architecture
Policy patterns that actually work
Scaling and isolation strategies
Observability and audit
Rollouts and versioning
A production checklist
References

When stdio stops being enough

MCP supports multiple transports; stdio is common for local servers. [2] In that model, the host controls process lifetime and secrets typically come from the environment on the local machine.

Stdio starts to strain when you need:

multi-client concurrency
shared tenancy
central policy enforcement
centralized audit
fleet-level rollout controls

At that point, you’re effectively building a platform. The platform needs a stable ingress point with consistent security and operational behavior.

MCP’s HTTP-based transports (like Streamable HTTP) are designed for servers that can handle multiple connections and enable streaming/notifications. [2] MCP also defines an authorization flow for HTTP-based transports. [3]

That’s the entry point for a gateway.

The MCP Gateway Pattern

Definition: An MCP gateway is an MCP server (or MCP-adjacent ingress layer) that:

authenticates and authorizes the client
routes requests to one or more downstream MCP servers (or tool backends)
enforces budgets and safety gates
emits consistent telemetry and audit records

It looks like an API gateway, but the payload is “tool capability” not “REST endpoints.”

Responsibilities of a gateway

1) Authentication and authorization

If you expose MCP servers over HTTP, you need strong auth. MCP includes an authorization framework at the transport layer for HTTP-based transports. [3]

Practical gateway rules:

Authenticate every client (bearer tokens, mTLS, OAuth-derived access tokens).
Authorize per tool, not per server.
Prefer least privilege scopes:
calendar.read
calendar.write
email.read
email.send
k8s.readonly
k8s.apply
For high-impact tools: require explicit confirmation tokens and/or multi-party approval.

2) Tool contract enforcement

MCP tools are invoked by an LLM-driven client. That means tool arguments are untrusted.

The gateway is the ideal place to enforce:

schema validation
payload size caps
allowlists and blocklists
“danger gates” (preview/apply, confirmations)
“semantic validation” (not just types - e.g., limits required, date ranges bounded)

MCP’s spec is grounded in structured schemas; treat those schemas as contracts. [1]

3) Budgets and backpressure

Agents can trigger bursty tool calls. Without backpressure you get the classic cascade:

upstream rate limits
DB pool exhaustion
thread/goroutine explosion
timeouts everywhere

At the gateway you can enforce:

per-tenant rate limits
per-tool concurrency limits
timeouts and deadline propagation
queue depth caps (bounded memory)
circuit breakers for flaky dependencies

This is where you keep “one user spamming tools” from becoming “everyone is down.”

4) Secret handling and redaction

Gateways are a natural place to centralize:

secret injection (short-lived tokens per tenant)
output redaction (strip tokens, emails, PII fields)
logging policies (never log raw tool payloads by default)

For agent systems, OWASP highlights risks like prompt injection and sensitive info disclosure as major categories. [7]

Your gateway should assume that anything returned by a tool could be coerced into exfiltration if you’re careless.

5) Observability and audit

Operationally, the gateway is your best place to emit consistent:

request logs
tool call metrics
traces across tool chains
audit events for side effects

OpenTelemetry is the de facto standard for collecting and exporting telemetry. [5] W3C Trace Context defines headers like traceparent/tracestate for trace propagation across services. [6]

If you want an enterprise to trust agents, you need the forensic trail.

6) Routing and discovery at scale

The gateway becomes:

the routing table (“tool X lives in cluster Y”)
the discovery system (“list tools available for tenant Z”)
the version broker (“tool schema v3 for client A, v4 for client B”)

This is also where you can implement “tool quality” policies:

quarantine tools with high error rates
fallback to read-only alternatives
degrade gracefully under partial outages

Reference architecture

Here’s a simple, effective gateway architecture:

--------------------------------
- Agent host / IDE / runtime -
- (MCP client) -
--------------------------------
 - Streamable HTTP / JSON-RPC [2][4]
 v
------------------------------------------------
- MCP Gateway -
- - AuthN/Z [3] -
- - Schema + safety gates -
- - Budgets (rate, concurrency, cost) -
- - Audit + telemetry (OTel) [5][6] -
- - Routing + tool registry -
------------------------------------------------
 -
 ------------------------
 v v
----------------- ------------------
- MCP Server A - - MCP Server B -
- (calendar) - - (k8s, github...)-
------------------ ------------------
 v v
 Upstream APIs Upstream APIs

Key design decision: the gateway should not contain business logic. It enforces policy and routes tool calls. Tool semantics live in tool servers.

Policy patterns that actually work

Pattern: Read vs write tool classes

Classify tools into tiers:

Read-only: listing, searching, fetching
Write-safe: creates/updates that are naturally reversible
Dangerous: deletes, bulk updates, destructive actions, privileged ops

Then enforce different rules per tier:

Read-only: wide availability, higher concurrency
Write-safe: lower concurrency, stronger audit, idempotency keys
Dangerous: preview/apply, explicit confirmations, restricted scopes

Pattern: Preview -> Apply

For any tool that can cause harm:

plan_* returns a plan + summary + plan_id
apply_* requires plan_id (and optionally a user confirmation token)

This is the “terraform plan/apply” mental model applied to tools.

Pattern: Allowlisted egress (SSRF containment)

If tools can fetch URLs or call arbitrary endpoints, treat it as SSRF risk. OWASP’s SSRF prevention guidance is a useful baseline. [8]

At the gateway, enforce:

allowlisted domains
IP/CIDR blocks for internal metadata ranges
redirect re-validation

Pattern: Tenant-bound tokens

Instead of giving tool servers “global” credentials, mint tenant-scoped tokens and inject them for each call.

reduces blast radius
makes audit meaningful
enables “kill switch” revocation per tenant

Scaling and isolation strategies

A gateway is where multi-tenancy becomes real. Choose an isolation model:

Option A: Process isolation per tool server (simple, strong isolation)

each integration is its own process/container
faults stay contained
rollouts per integration are easy

Tradeoff: more processes to manage.

Option B: Shared server with strong tenant sandboxing

single multi-tenant server handles many clients
cheaper to run
requires rigorous isolation inside the process

Tradeoff: higher risk if a bug leaks across tenants.

Option C: Hybrid

“sensitive” integrations are isolated
“low-risk” read-only tools can be multi-tenant

Most enterprises end up here.

Observability and audit

What to emit (minimum viable)

Metrics

tool_calls_total{tool, tenant, status}
tool_latency_ms{tool}
rate_limited_total{tenant}
budget_exceeded_total{tenant, budget_type}

Traces

request span (client -> gateway)
tool execution span (gateway -> server)
downstream spans (server -> upstream API)

Audit events

who (tenant/user/client)
what (tool + summarized parameters)
when
result (success/failure)
side effect IDs (resource IDs, plan_id, idempotency_key)

OpenTelemetry’s Go docs are a good reference for instrumentation patterns. [5]

Rollouts and versioning

Tool contracts drift. Clients upgrade at different times. Gateways can reduce pain by:

pinning tool schema versions per client
supporting additive changes first (new fields optional)
allowing parallel tool versions for a period
enabling canary rollouts per tenant

If you do nothing else: never deploy a breaking tool change to 100% of tenants at once.

A production checklist

Security

AuthN required for all HTTP-based access. [3]
AuthZ enforced per tool (least privilege).
Tool inputs validated and bounded.
Dangerous tools require preview/apply and explicit confirmations.
Egress allowlists exist for URL/network tools. [8]

Reliability

Per-tenant rate limiting and per-tool concurrency caps.
Timeouts everywhere; deadlines propagate.
Bounded queues (no unbounded memory growth).
Circuit breakers for flaky dependencies.

Operability

Traces propagate end-to-end (W3C Trace Context). [6]
Metrics and logs are consistent and redacted.
Audit events exist for side effects.

Delivery

Tool schemas versioned; canary rollouts supported.
Quarantine and fallback policies exist for failing tools.

References

[1] Model Context Protocol (MCP) - Specification (Protocol Revision 2025-11-25): https://modelcontextprotocol.io/specification/2025-11-25 [2] MCP - Transports (including Streamable HTTP): https://modelcontextprotocol.io/specification/2025-03-26/basic/transports [3] MCP - Authorization (HTTP-based transports): https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization [4] JSON-RPC 2.0 Specification: https://www.jsonrpc.org/specification [5] OpenTelemetry Go - Instrumentation docs: https://opentelemetry.io/docs/languages/go/instrumentation/ [6] W3C - Trace Context: https://www.w3.org/TR/trace-context/ [7] OWASP - Top 10 for Large Language Model Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/ [8] OWASP - SSRF Prevention Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html

Tool Discovery at Scale: Solving the Million Tool Problem

Sat, 15 Nov 2025 12:00:00 -0500

Why this matters

Tool-using agents are powerful because they can do real work: read systems, change systems, orchestrate workflows.

The trap is what I call the Million Tool Problem:

The moment you have “enough tools,” tool selection becomes harder than tool execution.

At small scale, you can stuff tool schemas into the prompt and hope the model chooses correctly. At scale, that approach breaks:

token budgets explode
accuracy drops (models confuse similar tools)
latency rises (bigger prompts, more reasoning)
safety degrades (wrong tool, wrong args, wrong side effects)

This isn’t hypothetical. Tool-use research exists because selection is hard. Benchmarks like ToolBench and AgentBench exist specifically to evaluate this capability in interactive settings. [3][6]

This post is a production-first design for tool discovery that stays:

fast (low latency, bounded prompt size)
safe (tool contracts and policy gates)
debuggable (you can explain why a tool was chosen)
maintainable (tool catalogs evolve constantly)

TL;DR

Tool discovery is an IR problem + a policy problem, not a prompt trick.
Use a 3-stage selector:

coarse filter (tags / domain / allowlist)
retrieval (BM25 + embeddings)
rerank (LLM or learned ranker)

Treat tool descriptions as a product:
consistent naming
sharp “when to use” / “when not to use”
examples of correct arguments
Add tool quality scoring (latency, error rate, drift, safety incidents).
Build a tight evaluation harness (ToolBench/StableToolBench ideas apply). [3][4]

Why “include all tools” fails
The 3-stage tool selector
Tool metadata that makes models smarter
Ranking: BM25 + embeddings + rerank
Safety: allowlists, “danger gates,” and budgets
Quality scoring and tool quarantine
Debuggability: explainable tool selection
A minimal reference architecture
A production checklist
References

Why “include all tools” fails

Token and latency pressure

Even if your tool schemas are “small,” they add up. Once you cross a few dozen tools, you spend more tokens describing tools than describing the task.

Confusability

Tools with similar names or overlapping domains cause selection errors:

search_events vs list_events vs get_event
create_task vs create_issue vs create_ticket

The long tail problem

Most catalogs have a long tail:

10 tools get used daily
100 tools get used weekly
1,000 tools are niche, but critical when needed

This is exactly the kind of situation information retrieval was invented for.

The 3-stage tool selector

Think like a search engine:

Stage 0: Policy filter (mandatory)

Before ranking, enforce policy:

which tools is this client allowed to call?
which tools are enabled for this tenant/environment?
which tools are safe for this context (read-only mode, incident mode, etc.)?

MCP makes tool discovery explicit via listing tools and schemas. That’s an interface you can mediate with policy. [1]

Stage 1: Coarse routing (cheap)

Route into the right “tool neighborhood” using:

tags (kubernetes, calendar, email)
domains (“devops”, “productivity”, “security”)
environment (“prod” vs “dev”)

Goal: reduce the candidate set from 10,000 -> 300.

Stage 2: Retrieval (BM25 + embeddings)

Run a hybrid search over:

tool name
tool description
parameter names
example calls
“when not to use” hints

Hybrid search is pragmatic:

lexical retrieval (BM25-style) is great for exact matches and acronyms [9]
embeddings are great for semantic similarity [7]

Goal: 300 -> 30.

Stage 3: Rerank (expensive, accurate)

Rerank the top-K tools using:

an LLM judge (cheap if K is small)
or a learned ranker
or deterministic rules + a smaller LLM tie-breaker

Goal: 30 -> 5.

Then the agent sees a small, high-quality tool set.

Tool metadata that makes models smarter

If you want better tool selection, stop treating tool schemas as “just types.” Add metadata that improves discrimination.

Tool card fields (recommended)

Name: stable, verb-first
Purpose: one sentence
When to use: 2-4 bullets
When NOT to use: 2-4 bullets (this is underrated)
Side effects: none / read-only / creates / updates / deletes
Required arguments: and why they’re required
Examples: 2-3 example invocations with realistic args
Error modes: rate limit, auth, not found, validation

This reduces tool confusion dramatically because it gives the model differentiating features.

Ranking: BM25 + embeddings + rerank

Lexical retrieval (BM25)

BM25 and probabilistic retrieval approaches are foundational in search. [9]

Practical benefit: it handles queries like:

“S3”
“JWT”
“PodDisruptionBudget”
“Cron” …where embeddings can be inconsistent.

Embeddings

Sentence embeddings (like SBERT-style approaches) are designed to enable efficient semantic similarity search. [7]

Practical benefit: it handles intent queries like:

“delete all tasks due tomorrow”
“find calendar conflicts next week”
“check if deployment is stuck”

Approximate nearest neighbor indexing

At scale, you’ll want ANN indexing (FAISS is a well-known library in this space). [8]

Rerank

This is where you incorporate:

tool quality score
tenant policy
“danger tool” gating
recent tool drift

Reranking is also where you can enforce “don’t pick write tools unless necessary.”

Safety: allowlists, “danger gates,” and budgets

Tool discovery is not neutral. It’s an authorization problem.

Your selector should be policy-aware:

Read-only mode: only surface read tools
No-delete mode: deletes never appear
Prod incident mode: allow observation tools, restrict mutation
Human approval mode: show write tools, but require confirmation

Also: build budgets into selection. If a tool is expensive (slow, rate-limited, high blast radius), rank it lower unless strongly justified.

For tool-using agents, OWASP highlights prompt injection and excessive agency as key risks - exactly the failure modes you get when tools are over-exposed without gates. [10]

Quality scoring and tool quarantine

You need a tool quality score because tools drift:

upstream APIs change
auth breaks
quotas shift
tool server regressions happen

Track per tool:

p50 / p95 latency
error rate
timeout rate
“invalid argument” rate (often a selection problem)
“unsafe attempt” rate (policy violations)

Then take action:

quarantine tools with regression spikes
degrade to read-only tools during outages
route to backups (alternate implementations)

Debuggability: explainable tool selection

If you can’t answer “why did the agent pick that tool?”, you won’t be able to operate the system.

Log (or attach to traces) the selection evidence:

query text
candidate tools (top 30)
retrieval scores
rerank scores
policy filters applied
final selected tools and why

This also becomes training data later.

A minimal reference architecture

-------------------------------
- Agent runtime (planner) -
-------------------------------
 -
 v
-------------------------------
- Tool Selector Service -
- - policy filter -
- - hybrid retrieval -
- - rerank -
- - tool quality weighting -
-------------------------------
 - returns top-K tools + schemas
 v
-------------------------------
- Agent execution -
- - calls tools via MCP -
-------------------------------

Where MCP fits: MCP provides a standardized way for clients to discover tools and invoke them. [1]

The selector doesn’t replace MCP. It makes MCP usable at scale.

A production checklist

Tool catalog hygiene

Stable naming conventions.
“When NOT to use” bullets exist.
Examples exist for the top tools.
Tool side effects are classified.

Selection pipeline

Mandatory policy filter before ranking.
Hybrid retrieval (lexical + embeddings). [7][9]
Rerank top-K with quality + policy.
Candidate set bounded (K is small).

Safety

Dangerous tools are gated and not surfaced by default.
Budget-aware ranking exists.
OWASP LLM risks considered in tool exposure strategy. [10]

Operability

Selection decisions are explainable (log evidence).
Tool quality scoring exists and drives quarantine.
Selection regressions are covered by evals (next article).

References

[1] Model Context Protocol (MCP) - Specification (Protocol Revision 2025-11-25): https://modelcontextprotocol.io/specification/2025-11-25 [2] MCP - Transports (including stdio and Streamable HTTP): https://modelcontextprotocol.io/specification/2025-03-26/basic/transports [3] ToolLLM / ToolBench (tool-use dataset + evaluation): https://arxiv.org/abs/2307.16789 [4] StableToolBench (stable tool-use benchmarking): https://arxiv.org/abs/2403.07714 [5] tau-bench (tool-agent-user interaction benchmark): https://arxiv.org/abs/2406.12045 [6] AgentBench (evaluating LLMs as agents): https://arxiv.org/abs/2308.03688 [7] Sentence-BERT (efficient semantic similarity search via embeddings): https://arxiv.org/abs/1908.10084 [8] FAISS / Billion-scale similarity search with GPUs: https://arxiv.org/abs/1702.08734 and https://github.com/facebookresearch/faiss [9] Robertson (BM25 and probabilistic relevance framework): https://dl.acm.org/doi/abs/10.1561/1500000019 [10] OWASP - Top 10 for Large Language Model Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/

The Service Template That Prevents Incidents

Sat, 25 Oct 2025 12:00:00 -0500

Why this matters

Most enterprises try to standardize software delivery with:

PDFs
Confluence pages
slide decks
architecture review boards

It doesn’t scale.

Teams don’t move faster because the rules exist. Teams move faster because the defaults exist.

Platform engineering language captures this well: paved roads / golden paths reduce cognitive load and make the “right way” the easy way. [1][2] The CNCF Platforms White Paper makes the case for internal platforms as a lever that impacts value streams indirectly - through better flow and developer experience. [3]

This article is a practical blueprint for the thing that actually changes outcomes:

A service template that bakes reliability, security, and operability into day-one defaults.

TL;DR

Build one paved road for APIs:
repo template + CI pipeline + runtime defaults
Include “boring” but critical capabilities:
health probes, resource requests/limits, disruption budgets [4][5][6]
tracing/metrics/logging via OpenTelemetry [7]
timeouts, retries, rate limits
standardized deployment and rollout
Measure success with outcomes (DORA metrics): lead time, deploy frequency, change failure rate, MTTR. [8]
Optimize for day 2 to day 50, not just “hello world.”

What a paved road is (and isn’t)
The API service template: required capabilities
A reference repository structure
Kubernetes defaults that save you later
Observability by default
Security by default
Rollouts and operational controls
How to roll this out without a platform revolt
A production checklist
References

What a paved road is (and isn’t)

A paved road is

a recommended path to production
preconfigured defaults that make safe delivery easy
automation that eliminates repetitive decisions

Microsoft describes this in internal developer platform terms: recommended and supported development paths, incrementally paved through an internal platform. [2]

A paved road is not

a mandate that blocks all other approaches
a committee process
a doc nobody reads

If your paved road becomes a gate, teams will route around it.

The API service template: required capabilities

Here’s what “enterprise production API” should mean out of the box.

Operability

structured logging with correlation IDs
metrics (request rate/latency/errors)
tracing across inbound/outbound calls [7]
runtime config and feature flags

Reliability

timeouts everywhere
bounded retries with backoff
health probes (liveness/readiness/startup) [5]
graceful shutdown
rate limits / concurrency caps

Platform fit

Kubernetes-ready manifests
resource requests/limits [4]
PodDisruptionBudget for availability during maintenance [6]
standardized rollout strategy

Security

auth middleware
input validation
secret injection patterns (no secrets in repo)
least privilege service accounts

Delivery

CI pipeline: lint/test/build/scan
SBOM generation
deploy automation (GitOps or pipeline)

A reference repository structure

.
--- cmd/service/ # main
--- internal/ # business logic
--- pkg/ # shared libs (optional)
--- api/ # OpenAPI spec, schemas
--- deploy/
- --- k8s/ # manifests (or Helm/Kustomize)
- --- policy/ # OPA/constraints (optional)
--- docs/
- --- index.md
- --- runbooks/
--- Makefile
--- .github/workflows/ # CI

Key idea: the template is not just code - it is the full production story:

how to run locally
how to deploy
how to observe
how to operate on-call

Kubernetes defaults that save you later

1) Resource requests and limits

Kubernetes scheduling and stability depend on requests/limits. The official docs explain how pod requests/limits are derived from container values. [4]

Template default:

set conservative requests
set safe limits
provide guidance for right-sizing

2) Probes

Kubernetes supports liveness, readiness, and startup probes. The docs describe how to configure them and why they matter. [5]

Template default:

readinessProbe ensures traffic only goes to ready pods
livenessProbe catches deadlocks / stuck processes
startupProbe prevents early restarts for slow boot services

3) Disruption budgets

PodDisruptionBudgets limit concurrent disruptions during voluntary maintenance. [6]

Template default:

include a PDB for replicated services
define min available or max unavailable

Observability by default

If you do one thing: instrument the template so every service ships with telemetry.

OpenTelemetry provides the framework for standard traces/metrics/logs. [7]

Template defaults:

standard HTTP server instrumentation
propagation of trace context (W3C headers)
request logs include trace IDs
golden dashboard:
RPS
p95 latency
error rate
saturation (CPU/memory)

Security by default

Avoid “security guidance documents.” Make secure defaults.

Template defaults:

auth middleware with standardized claims/roles mapping
structured validation for request bodies
outbound allowlists (where feasible)
secret injection via environment/secret store (no plain text)

Your paved road becomes a security accelerator because teams start secure.

Rollouts and operational controls

Default rollout patterns:

canary or progressive delivery when needed
safe rollback
feature flags for risky changes

Default operational controls:

rate limiting
concurrency limits
timeouts and circuit breakers
“maintenance mode” toggle

How to roll this out without a platform revolt

This is the part platform teams often miss.

1) Make it optional - but obviously better

If adopting the template reduces weeks of work to hours, teams will choose it.

2) Provide migration paths

minimal adoption: observability + probes
medium: deploy manifests + CI
full: service template + libraries

3) Measure outcomes, not adoption

Use DORA metrics to show impact: lead time, deploy frequency, change failure rate, time to restore service. [8]

If the paved road doesn’t move these, it’s not paved.

A production checklist

Template

Repo template includes CI, deploy, docs, runbooks.
Observability instrumentation included by default. [7]

Kubernetes

Resource requests/limits included. [4]
Liveness/readiness/startup probes included. [5]
PodDisruptionBudget included for replicated services. [6]

Reliability

Timeouts and bounded retries are standard.
Graceful shutdown is implemented.
Rate limiting/concurrency caps exist.

Security

Auth middleware included.
Secrets handled via secure injection (not repo).

Outcomes

DORA metrics tracked to validate improvement. [8]

References

[1] CNCF - What is platform engineering? (golden paths/paved roads framing): https://www.cncf.io/blog/2025/11/19/what-is-platform-engineering/ [2] Microsoft Learn - What is platform engineering? (paved paths / internal developer platform): https://learn.microsoft.com/en-us/platform-engineering/what-is-platform-engineering [3] CNCF TAG App Delivery - Platforms White Paper: https://tag-app-delivery.cncf.io/whitepapers/platforms/ [4] Kubernetes - Resource Management for Pods and Containers (requests/limits): https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ [5] Kubernetes - Configure Liveness, Readiness and Startup Probes: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ [6] Kubernetes - Specifying a Disruption Budget for your Application (PDB): https://kubernetes.io/docs/tasks/run-application/configure-pdb/ [7] OpenTelemetry - Documentation (instrumentation and telemetry): https://opentelemetry.io/docs/ [8] DORA - DORA’s software delivery performance metrics: https://dora.dev/guides/dora-metrics/

Go MCP Server Ecosystem

Sun, 01 Sep 2024 00:00:00 +0000

Summary

This project is a growing ecosystem of Model Context Protocol (MCP) servers written in Go. Each server wraps a real service (calendar, email, task management, knowledge base, etc.) and exposes it as a typed, tool-based interface for MCP clients (e.g., Claude Desktop / Claude Code). [1][2]

The theme is simple: agents are only as useful as the tools they can call, and “tooling” needs the same production bar as any other integration layer: security boundaries, backpressure, observability, and predictable failure modes.

Open-source MCP servers

iCloud Calendar MCP Server (CalDAV): list calendars, search events, create/update/delete events; includes recurring event expansion, multi-account support, rate limiting, retries, audit logs, and Prometheus/health endpoints. [4]
iCloud Email MCP Server (IMAP/SMTP): search and read mail, send/reply, manage folders, handle attachments, and apply safety annotations (read-only vs destructive) with strict input validation. [5]
Todoist MCP Server (REST API v2 + Sync batching): manage tasks/projects/labels/comments; supports bulk operations with rate-limit-aware batching and Todoist filter syntax. [6]
Notion MCP Server (Notion REST API): pages, databases, blocks, comments, users; includes templates, exports (Markdown/CSV), smart queries, and built-in throttling/retries. [7]

Private / not-yet-open-sourced connectors

I’ve also built MCP connectors for enterprise systems that aren’t ready to open-source yet (either due to org-specific assumptions, credentials, or hard-coded domain models):

Kubernetes
Argo CD
SonarQube
GitHub
Temporal
OpenText Octane

(These follow the same design patterns described below.)

Problem

Agents need to interact with real systems: calendars, email, task systems, and internal developer platforms. Without a standard interface, every tool integration becomes a one-off, and reliability/guardrails drift between projects.

MCP solves the “standard interface” problem by defining how a host/client can discover and call server-exposed tools over a consistent protocol. [1][2] This ecosystem focuses on solving the remaining hard part: making those integrations production-grade.

Constraints

Local-first security boundary: credentials live on the host where the server runs (env vars, secret mounts, keychain tooling); the server talks directly to the upstream service with no proxy SaaS. [4][5]
Safety by design: explicit tool schemas, input validation, and tool classification (read-only vs mutating) so clients can apply guardrails. [4][5]
Fast & predictable: low startup time and bounded tool-call latency (timeouts + backpressure). [4][5][7]
Operable like a real service: logs that correlate per request, rate limiting, retries/backoff where appropriate, and health/metrics where it matters. [4][6][7]
Portable distribution: ship as single Go binaries (and containers where useful), so the “tool layer” is easy to deploy alongside agents. [4][5]

Architecture

At a high level, every server follows the same pattern:

MCP client (hosted by Claude / an agent runtime) communicates with the server (typically over stdio transport).
The server validates inputs, applies middleware (timeouts, logging, rate limits), and calls the upstream API/protocol.
Results are mapped into safe, typed tool outputs (and errors are normalized for the client).

┌───────────────────────────┐ MCP (tools) ┌────────────────────────────┐
│ Claude Desktop / Code │ ───────────────────────────▶ │ mcp-<service> (Go binary) │
│ (MCP host + client) │ ◀─────────────────────────── │ - tool schemas + handlers │
└───────────────────────────┘ JSON-RPC/session │ - auth + validation │
 │ - rate limit + retries │
 └───────────┬────────────────┘
 │
 │ service protocol / API
 ▼
 ┌──────────────────────────────┐
 │ iCloud / Todoist / Notion ... │
 └──────────────────────────────┘

Cross-cutting “production traits”

Instead of building one-off scripts, these servers implement common production patterns:

Timeout middleware on every tool call (so agents don’t hang forever). [4][5][7]
Request correlation IDs and structured logs (debuggable across multi-step agent runs). [4][5]
Rate limiting + backoff when upstream services throttle (e.g., iCloud and Notion). [4][7]
Bulk operation strategies that reduce API calls (e.g., Todoist Sync API batching for bulk changes). [6]
Health + metrics endpoints where running in containers makes sense (notably the iCloud Calendar server). [4]
Automated CI (race detector, linting, vulnerability checks) to keep “tool servers” from becoming unreviewed glue. [4][5]

Key decisions

Go for tool servers: predictable concurrency, easy cross-platform builds, and the “single static-ish binary” deployment model fits MCP servers well, especially when they’re launched per-session or run as small sidecars. [4][5]
Independent binaries per integration: calendar ≠ email ≠ tasks. Separate processes isolate failures, limit blast radius, and make upgrades/rollbacks straightforward.
Local-first auth: app-specific passwords (iCloud), API tokens (Todoist), integration tokens (Notion). The servers are designed so secrets stay on your machine / in your cluster secrets manager, not copied into prompts. [4][5][6][7]
Use an MCP SDK, focus on semantics: the implementations use the Go MCP SDK (mark3labs/mcp-go) so most effort goes into tool behavior, validation, and safety. [8][9]

Outcome

This ecosystem has produced multiple MCP servers that are:

useful (real workflows: schedule management, inbox operations, task execution, knowledge base automation),
operationally hardened (timeouts, retries, rate limits, observability),
portable (binaries + releases for easy distribution),
and structured enough to be safe (typed schemas, validation, tool annotations).

Concrete examples from the current repos:

The iCloud Calendar server exposes 5 tools, supports multi-account, and includes health + Prometheus metrics, audit logging without PII, retries/backoff, and rate limiting. [4]
The iCloud Email server exposes 14 tools and includes thread-safe IMAP access, request correlation IDs, strict validation, and “read-only vs destructive” tool annotations. [5]
Tagged releases exist across the servers (e.g., iCloud Calendar v1.1.0, iCloud Email v0.6.0, Todoist v1.0.0, Notion v0.8.0 published on Feb 7, 2026). [4][5][6][7]

Stack

Go, MCP, mark3labs/mcp-go, CalDAV, IMAP/SMTP, REST APIs (Todoist/Notion), Docker (distroless where applicable), Prometheus metrics (where applicable).

References

[1] Model Context Protocol (MCP): Specification (Protocol Revision 2025-11-25). https://modelcontextprotocol.io/specification/2025-11-25 [2] Model Context Protocol (MCP): Architecture (Protocol Revision 2025-06-18). https://modelcontextprotocol.io/specification/2025-06-18/architecture [3] Roy Gabriel: “Go MCP Server Ecosystem” (original portfolio page). https://www.roygabriel.dev/projects/mcp-servers/ [4] GitHub: roygabriel/mcp-icloud-calendar. https://github.com/roygabriel/mcp-icloud-calendar [5] GitHub: roygabriel/mcp-icloud-email. https://github.com/roygabriel/mcp-icloud-email [6] GitHub: roygabriel/mcp-todoist. https://github.com/roygabriel/mcp-todoist [7] GitHub: roygabriel/mcp-notion. https://github.com/roygabriel/mcp-notion [8] GitHub: mark3labs/mcp-go. https://github.com/mark3labs/mcp-go [9] go.mod (module dependencies) for the MCP servers (e.g., mark3labs/mcp-go used in this ecosystem). - https://raw.githubusercontent.com/roygabriel/mcp-icloud-calendar/main/go.mod - https://raw.githubusercontent.com/roygabriel/mcp-icloud-email/main/go.mod - https://raw.githubusercontent.com/roygabriel/mcp-todoist/main/go.mod - https://raw.githubusercontent.com/roygabriel/mcp-notion/main/go.mod

Go | Roy Gabriel

Cruvero - AI Agent Ecosystem Platform

Summary

The Problem

Architecture

Neuro-Inspired Intelligence

Enterprise Governance

Tool Ecosystem & MCP Integration

Observability & Operations

Key Decisions

Outcome

Stack

Go vs Spring Boot for Enterprise APIs: Cost, Performance, and Cloud-Native Ops

Why this comparison keeps showing up

TL;DR

Contents

The cost model: what you actually pay for

A simple (illustrative) density example

Go’s production advantages (when they matter)

1) Packaging simplicity and deployment surface

2) Fast start and “scale events”

3) Resource envelope predictability

4) Cloud-native ergonomics: minimalism wins over time

Where Spring Boot is still the right tool

1) Ecosystem and “starter” leverage

2) JVM performance can be excellent

3) Operational maturity and tooling

Cloud-native reality: images, CVEs, and deploy surface

Distroless is not a Go-only advantage

Two Dockerfile patterns (illustrative)

Observability and operations

OpenTelemetry support

Java’s advantage: zero-code instrumentation

Go’s reality: explicit instrumentation (plus growing options)

A decision matrix

How to validate with a real experiment

1) Define a representative endpoint mix

2) Measure the four numbers that matter

3) Run the same load and failure profile

4) Compare operational work, not only performance

Common failure modes

Go pitfalls

Spring Boot pitfalls

Both ecosystems

Closing thought

References

MCP Servers in Production: Hardening, Backpressure, and Observability (Go)

Why this matters

TL;DR

Contents

A production mental model for MCP servers

Threat model: what actually goes wrong

1) Input ambiguity → destructive actions

2) Prompt injection → tool misuse

3) SSRF / network pivoting

4) Unbounded concurrency → resource collapse

5) “Helpful logs” → data leak

Hardening layer 1: identity and authorization

Practical rules

Go pattern: a minimal auth middleware skeleton (HTTP transport)

Hardening layer 2: tool contracts that resist ambiguity

Design tools like production APIs

Add a “preview → apply” flow for risky tools

Hardening layer 3: budgets and backpressure

Budget checklist

Go: server timeouts are not optional

Go: propagate cancellation everywhere with context

Go: per-tenant rate limiting with x/time/rate

Backpressure: choose a policy

Hardening layer 4: safe networking and SSRF containment

SSRF containment strategies that actually work

Go pattern: an outbound HTTP client with strict timeouts

Hardening layer 5: observability without leaking secrets

What to measure (minimum viable MCP telemetry)

Trace boundaries

Logging rules that save you later

Hardening layer 6: versioning and rollout discipline

Practical compatibility rules

Rollout like a platform team

A production checklist

Go: propagate cancellation everywhere with `context`

Go: per-tenant rate limiting with `x/time/rate`