Roy Gabriel - DevOps Architect & Applied AI Engineer | Roy Gabriel

Agent Swarms: The New Workforce Architects and Engineers Must Lead

Thu, 26 Feb 2026 13:00:00 -0500

As-of note: This is a production engineering perspective based on building and running agent swarms in production in early 2026. The workflows, numbers, and fine-tuning example come from real runs on my hardware and standards.

Why this shift keeps showing up

If you build production systems long enough, you see the same pattern:

The “tool choice” is not what breaks velocity.
The operational model and who does the work usually are.

When architects and engineers compare traditional coding to agent swarms, they are really asking:

“What will it cost to ship features at scale in the next 12 to 36 months, and who will actually build them?”

Single agents autocomplete code. Agent swarms plan features, assign tasks, review security, run tests, and deploy. The output is cleaner and more consistent than what most senior engineers produce alone.

Anthropic’s 2026 Agentic Coding Trends Report shows single agents evolving into coordinated teams that handle long-running tasks across the full development lifecycle. Multi-agent orchestration is the breakthrough.

Forbes reported in December 2025 that Gartner expects 40 percent of enterprise applications to embed task-specific agents by the end of 2026. Orchestrated swarms already deliver 30 to 50 percent faster feature delivery in early adopters.

O’Reilly’s Signals for 2026 confirms the pattern: engineers move from writing code to orchestrating agents. Fundamentals stay essential. The difference is scale.

TL;DR

Agent swarms already build production code at higher quality and speed than the top 1 percent of FAANG engineers.
The new job is writing precise specs, designing the agent workforce, and verifying outcomes.
I built Cruvero exactly for this model. It is the production control plane that manages swarms in production.
In one run I used Cruvero to prep my entire personal knowledge base and fine-tune Qwen2.5-Coder-72B on two RTX 5090 GPUs. The resulting model now writes code exactly to my standards.
Cruvero will be released as open source in the next two weeks so anyone can run the same production-grade agent swarm control plane.
Start today or watch the gap widen.

What Agent Swarms Do Today
Cruvero as the Working Example
The New Role for Architects and Engineers
Why You Must Start Today
References

What Agent Swarms Do Today

A single agent writes a function. A swarm plans an entire feature, breaks it into tasks, assigns roles, iterates on failures, runs tests, and deploys. The output is cleaner, more consistent, and more reliable than what most senior engineers produce alone.

Multi-agent orchestration lets one agent plan, another implement, a third review for security, and a fourth measure performance. They collaborate without human prompts for hours or days.

Cruvero as the Working Example

Over the last 2 months on weekends and nights, I built Cruvero as a full workflow builder and the complete DevOps/SRE agent swarm platform. It is 350k lines of clean Go and React running as a production control plane, with a clean UI and API layer for knowledge bases, multi-agent runs, agent registration, cost tracking, flows, and MCP bindings in Kubernetes. In early 2026, I have not seen another platform that ships this full stack in one place.

Core capabilities include:

Knowledge bases that store versioned domain context so agents never repeat mistakes
Runs page for launching, monitoring, and intervening in complex multi-agent workflows with patterns like Delegate
Agents page that lets you register, trust-score, deploy, and bind agents directly from the UI
Incident triage that runs faster than human response, then posts root-cause analysis and recommended fixes directly into Slack or Microsoft Teams
Pre-run previews, compliance checks, per-agent overrides, full audit trails, and cost tracking

This turns raw agent swarms into a manageable, auditable workforce. This week the platform reached the point where its own agent swarms are building new features inside Cruvero itself.

In one recent run, a swarm of 12 agents took a high-level spec for a new MCP gateway, generated the Go handlers, added observability, wrote tests, and produced a Helm chart ready for deployment. The entire process took 47 minutes. I reviewed the plan, approved two overrides, and signed off. The code passed every gate I set.

Here is a concrete example of how deeply I integrated my own standards. This week I used Cruvero to create a domain-specific fine-tuned model. I pointed the swarm at my private knowledge base: every personal(not employer code) code example I have written in the last five years, my strict documentation standards (every public function must include invariants, performance notes, and failure modes in a precise JSDoc format), my architecture standards (hexagonal with CQRS and explicit boundaries), and my full infrastructure layout (multi-region Kubernetes with Cilium CNI, ArgoCD GitOps, external secrets, and cross-cluster service mesh using Istio).

The swarm prepped the entire dataset in 18 minutes. It extracted, cleaned, deduplicated, and converted everything into instruction-response pairs that perfectly match the way I write and review code. Then it launched a QLoRA fine-tune of Qwen2.5-Coder-72B across my two RTX 5090 GPUs. The training completed overnight. The resulting model, now called Cruvero -Gabriel-72B, is registered directly inside the agent swarm as a first-class capability.

When I give the swarm a new spec today, it calls my fine-tuned model for every code generation step. The output follows my exact coding style, documentation format, architecture patterns, and infrastructure conventions without any manual correction. The swarm literally thinks and builds like I do, at 100 times the speed.

Cruvero will be released as open source in the next two weeks so any engineer or team can self-host the same production-grade agent swarm control plane I use daily.

The New Role for Architects and Engineers

You will spend your time on three things:

Write precise specification documents. Define goals, constraints, acceptance criteria, security boundaries, and stop rules. These become the source of truth the swarm follows.
Design the agent workforce. Choose which agents handle planning, implementation, review, and verification. Set trust scores, capabilities, and escalation paths.
Verify and steer. Run preflight checks, inspect outputs against intent, and adjust the swarm in real time. Your judgment determines whether the final system meets business needs.

This is harder than writing code. It requires deep systems thinking, clear communication, and production discipline. The best architects already excel at it. The rest must learn fast.

Pragmatic Engineer noted in January 2026 that teams where AI writes 90 percent of the code still need senior engineers for architecture decisions and coherence. Karpathy has said the same: technical expertise becomes more valuable because skilled people extract far more from agents.

Why You Must Start Today

The gap between those who manage agent swarms and those who do not will widen quickly. In 12 months, the best teams will ship features that used to take quarters. In 36 months, organizations without swarm capability will struggle to compete on velocity and quality.

Start small. Take one workflow you own. Break it into phases. Build a prompt library. Run a swarm on a non-critical task. Measure the output against your own work. Iterate.

The tools exist now: Claude Code, Codex, open MCP gateways, and platforms like Cruvero . The question is whether you will lead the workforce or watch someone else do it.

The next generation of software systems will be built by agent swarms under human direction. Architects and engineers who master this model will define the future. Those who wait will find their skills replaced.

The shift is here. Lead it.

References

Anthropic. 2026 Agentic Coding Trends Report. January 2026.
Minevich, Mark. “Agentic AI Takes Over — 11 Shocking 2026 Predictions.” Forbes. December 31, 2025.
O’Reilly Media. “Signals for 2026.” January 9, 2026.
Various sources on Karpathy statements compiled in Pragmatic Engineer and The New Stack, 2025–2026.

Chapter 16: Worked Example: Converting an Ansible Playbook to a Go Temporal Workflow

Fri, 13 Feb 2026 09:00:00 -0500

Series: LLM Development Guide

Chapter 16 of 16

Previous: Chapter 15: Worked Example: Creating a Helm Chart From a Reference Chart

What you’ll be able to do

You’ll be able to migrate a procedural automation (for example, an Ansible playbook) into a durable Temporal workflow:

Extract discrete steps from the playbook.
Map steps to activities.
Implement a workflow that follows your team’s existing Temporal patterns.
Add verification and tests so the migration is not faith-based.

TL;DR

Start from a working reference workflow in your repo.
Paste both the playbook and the reference into the planning prompt.
Define activities first, then the workflow.
Verify with Temporal workflow tests and any integration checks you can run safely.

Scenario
Reference inputs
Plan and phase structure
Implementation skeleton (Go)
Verification
Gotchas

Scenario

Example: convert a playbook that creates a Kubernetes namespace and resource quota into a Temporal workflow.

Why this is a good fit:

Work is step-based.
You want retries and observability.
You want an execution history.

Reference inputs

Paste these into your planning prompt:

The playbook file (or the relevant section): playbooks/create-namespace.yml.
A reference workflow file that represents your team’s patterns.
A reference activity implementation file (if you have one).

Suggested commands:

sed -n '1,200p' playbooks/create-namespace.yml

sed -n '1,200p' internal/workflows/provision_cluster.go

ls -la internal/activities || true

Plan and phase structure

A reasonable phase split:

Phase 1: define activity I/O and implement activities.
Phase 2: implement workflow with retries and timeouts.
Phase 3: add tests.
Phase 4: register and deploy.

The plan should include verification per phase.

Implementation skeleton (Go)

This is a minimal skeleton to illustrate structure. It is intentionally not a full program.

package workflows

import (
 "time"

 "go.temporal.io/sdk/temporal"
 "go.temporal.io/sdk/workflow"
)

type CreateNamespaceInput struct {
 Namespace string
 CPULimit string
 MemLimit string
}

type CreateNamespaceOutput struct {
 Namespace string
}

// CreateNamespaceWorkflow orchestrates namespace creation.
// It assumes there are activities registered for create + quota + verify.
func CreateNamespaceWorkflow(ctx workflow.Context, input CreateNamespaceInput) (*CreateNamespaceOutput, error) {
 logger := workflow.GetLogger(ctx)
 logger.Info("starting namespace workflow", "namespace", input.Namespace)

 ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
 StartToCloseTimeout: 10 * time.Minute,
 RetryPolicy: &temporal.RetryPolicy{
 MaximumAttempts: 3,
 },
 })

 // Pseudocode: adapt to your activity names and input/output types.
 // var nsResult activities.CreateNamespaceOutput
 // err := workflow.ExecuteActivity(ctx, activities.CreateNamespace, activities.CreateNamespaceInput{...}).Get(ctx, &nsResult)
 // if err != nil { return nil, err }

 // var quotaResult activities.CreateResourceQuotaOutput
 // err = workflow.ExecuteActivity(ctx, activities.CreateResourceQuota, activities.CreateResourceQuotaInput{...}).Get(ctx, &quotaResult)
 // if err != nil { return nil, err }

 // var verifyResult activities.VerifyNamespaceOutput
 // err = workflow.ExecuteActivity(ctx, activities.VerifyNamespace, activities.VerifyNamespaceInput{...}).Get(ctx, &verifyResult)
 // if err != nil { return nil, err }

 return &CreateNamespaceOutput{Namespace: input.Namespace}, nil
}

Notes:

The workflow.ExecuteActivity calls are left as pseudocode because activity package names and types are repo-specific.
Keep the skeleton syntactically correct.
Use your reference workflow as the style guide.

Verification

Your verification should include at least one of these:

Temporal workflow unit tests (Temporal test framework).
Activity unit tests (mock external systems).
A safe integration test against a non-production environment.

Example commands (adapt to your repo):

# Run unit tests.
go test ./...

# If you have a focused workflow test package.
go test ./internal/workflows -run TestCreateNamespaceWorkflow

Expected results:

Tests exit with code 0.
Failures are actionable (not timeouts with no logs).

Gotchas

If you skip the reference workflow, your new workflow will not match team patterns.
If you skip tests, you will not know whether retries and timeouts behave correctly.
LLMs will happily invent Temporal APIs. Verify imports and method names exist in your actual SDK version.

Chapter 15: Worked Example: Creating a Helm Chart From a Reference Chart

Wed, 11 Feb 2026 07:00:00 -0500

Series: LLM Development Guide

Chapter 15 of 16

Previous: Chapter 14: Building a Prompt Library: Governance + Quality Bar

Next: Chapter 16: Worked Example: Converting an Ansible Playbook to a Go Temporal Workflow

What you’ll be able to do

You’ll be able to create a production-quality Helm chart by following an existing chart in your repo as the reference:

Gather high-signal reference inputs.
Produce a phased plan and prompt docs.
Execute in reviewable commits.
Verify the chart renders and lints cleanly.

TL;DR

The reference chart is the source of truth for structure and conventions.
Paste the reference inputs into your planning prompt.
Execute one file at a time, with helm lint and helm template as gates.
If you do not have a real reference chart, pick a different worked example.

Scenario
Reference inputs
Phase 1: Plan
Phase 2: Prompt docs
Phase 3: Execute in logical units
Verification
Gotchas

Scenario

Goal: create a new chart (example: metrics-gateway) based on a known-good reference chart (example: event-processor).

This is a workflow example. You will need to substitute your real chart names and paths.

Reference inputs

Run these commands in your repo and paste the output into the planning prompt.

# Chart structure and key files.
tree charts/event-processor/

sed -n '1,200p' charts/event-processor/Chart.yaml

sed -n '1,200p' charts/event-processor/values.yaml

sed -n '1,200p' charts/event-processor/templates/_helpers.tpl

# If your reference chart uses these, include them too.
ls -la charts/event-processor | rg -n "values-" || true

Why this matters:

Structure: avoids “generic Helm” output.
Naming and labels: keeps your charts consistent.
Values shape: keeps operator UX consistent.

Phase 1: Plan

Create a plan that is mostly:

What files will exist.
What differences are specific to metrics-gateway (ports, probes, resources).
How you will verify each phase.

Example plan skeleton:

# metrics-gateway Helm Chart Plan

## Goals
- Create charts/metrics-gateway matching reference structure.
- Render successfully with helm template.
- Lint cleanly.

## References
- charts/event-processor/

## Phase 1: Analysis
- [ ] Document naming conventions from reference.

Verification:
- tree charts/event-processor

## Phase 2: Scaffold
- [ ] Create Chart.yaml
- [ ] Create values.yaml
- [ ] Create templates/_helpers.tpl

Verification:
- helm lint charts/metrics-gateway

## Phase 3: Core templates
- [ ] deployment
- [ ] service
- [ ] configmap

Verification:
- helm template charts/metrics-gateway > /tmp/rendered.yaml

## Definition of done
- [ ] helm lint exits 0
- [ ] helm template exits 0

Phase 2: Prompt docs

Generate one prompt file per phase. Include:

The plan path.
The work-notes path.
Reference chart file paths.
Deliverables (exact files).
Constraints (MUST and MUST NOT).
Verification commands.

A good constraint to include:

“Match reference structure exactly.” (and name what that means)

Phase 3: Execute in logical units

You have two implementation strategies.

Strategy A (recommended): copy the reference chart, then adapt

This is often the fastest way to guarantee structure consistency.

cp -R charts/event-processor charts/metrics-gateway

# Then rename strings and values in a controlled way.
# Review each replacement before committing.
rg -n "event-processor" charts/metrics-gateway

Now execute in logical units:

Update Chart.yaml.
Update values.yaml.
Update _helpers.tpl.
Update one template file at a time.

For each logical unit:

Update work notes.
Run helm lint.
Propose a commit.

Strategy B: scaffold from scratch, guided by the reference

Use this when copying would bring too much baggage.

You still paste the reference files, but ask the model to reproduce the structure explicitly.

Verification

Run both linting and rendering.

helm lint charts/metrics-gateway

helm template charts/metrics-gateway > /tmp/metrics-gateway.rendered.yaml

test -s /tmp/metrics-gateway.rendered.yaml

Expected results:

All commands exit with code 0.
The rendered YAML is non-empty.

Optional: diff against the reference chart structure:

# Compare structure only.
(cd charts/event-processor && find . -type f | sort) > /tmp/ref-files.txt
(cd charts/metrics-gateway && find . -type f | sort) > /tmp/new-files.txt

diff -u /tmp/ref-files.txt /tmp/new-files.txt || true

Expected result:

The file lists are close, with only intentional differences.

Gotchas

If you do not paste the reference files, you will get generic charts.
Be explicit about service ports, probe paths, and resource defaults.
Add negative constraints (“do not add ingress yet”) so scope doesn’t expand.

Continue -> Chapter 16: Worked Example: Converting an Ansible Playbook to a Go Temporal Workflow

Chapter 14: Building a Prompt Library: Governance + Quality Bar

Mon, 09 Feb 2026 06:00:00 -0500

Series: LLM Development Guide

Chapter 14 of 16

Previous: Chapter 13: Templates + Checklists: The Copy/Paste Kit

Next: Chapter 15: Worked Example: Creating a Helm Chart From a Reference Chart

What you’ll be able to do

You’ll be able to build a prompt library that doesn’t turn into a junk drawer:

Organize prompts by task type.
Define a consistent prompt entry format.
Set a contribution and maintenance policy.

TL;DR

A prompt library is a shared collection of prompts proven in real usage.
Require prereqs, recommended model tier, expected output, and common failure fixes.
Assign maintainers.
Version prompts with a changelog.

Library structure
Prompt entry template
Contribution guidelines
Governance
Verification

Library structure

A simple layout that scales:

prompt-library/
 README.md
 CONTRIBUTING.md
 planning/
 implementation/
 testing/
 review/
 debugging/

Keep it boring. Avoid inventing new categories every week.

Prompt entry template

Require a consistent format so prompts are reusable:

# <Task Name>

## When to use

## Prerequisites
-

## Recommended model tier

## The prompt

## Customization points

## Expected output

## Common issues and fixes

## Examples

## Changelog
- YYYY-MM-DD: <what changed>

Contribution guidelines

Set a quality bar:

A prompt must have been used successfully multiple times.
It must specify required reference files.
It must include verification.
It must include common failure modes and fixes.

A contribution checklist:

Used successfully 3+ times.
Another person can run it with the listed prereqs.
Changelog updated.

Governance

If nobody owns it, it rots.

Assign 1 to 2 maintainers to:

Review new prompts.
De-duplicate similar prompts.
Archive prompts that no longer work.
Run a quarterly cleanup.

Verification

Bootstrap the skeleton:

mkdir -p prompt-library/{planning,implementation,testing,review,debugging}

touch prompt-library/README.md

touch prompt-library/CONTRIBUTING.md

cat > prompt-library/planning/new-task.md <<'MD'
# New Task Planning

## When to use

## Prerequisites

## Recommended model tier

## The prompt

## Verification
MD

Expected result:

You have a real place to put prompts that worked, with enough structure to keep it maintainable.

Continue -> Chapter 15: Worked Example: Creating a Helm Chart From a Reference Chart

When Enterprise Defaults Become Enterprise Debt

Sat, 07 Feb 2026 09:00:00 -0500

Note on examples: The scenarios below are anonymized composites. They’re not a critique of any one organization; they’re patterns that repeat across industries. The goal isn’t to “modernize for fun.” It’s to protect speed-to-market and reliability as systems and organizations scale.

Why this matters

Most enterprises don’t lose because they picked the “wrong” framework or cloud provider. They lose because old defaults - once rational - become invisible policy.

The 90s and early 2000s optimized for constraints that were real at the time:

hardware was expensive
automation was immature
environments were scarce
security controls were largely manual
uptime was achieved by cautious change, not by safe change

Those constraints have shifted. But many organizations still run on architectural and governance defaults designed for a different era.

The result is predictable:

innovation slows (lead time grows)
quality degrades (late integration + big-bang changes)
reliability suffers (risk is batched, blast radius expands)
engineers spend more time navigating the system than improving it

If you want a single sentence summary: old patterns don’t just slow delivery - they also create the conditions for outages.

TL;DR

Retire “analysis as delivery.” Timebox discovery and ship thin vertical slices.
Treat cloud primitives as primitives, not research projects (e.g., object storage is solved).
Default to containers + orchestration for most stateless services; use VMs deliberately, not reflexively. [5]
Replace ticket queues and boards with guardrails + paved roads + policy-as-code. [7][8]
Measure what matters: lead time, deploy frequency, change failure rate, MTTR. [1][2]
Modernization works best as an incremental program, not a rewrite (Strangler Fig pattern). [12]

Pattern 1: Analysis as a substitute for delivery
Pattern 2: Reinventing commodity infrastructure
Pattern 3: VM-first thinking as the default
Pattern 4: Ticket-driven infrastructure
Pattern 5: Change Advisory Board for routine changes
Pattern 6: The shared database empire
Pattern 7: Central integration as a chokepoint
Pattern 8: Perma-POCs and innovation theater
Replace committees with guardrails
Modernize without a rewrite
Verification: how you know it’s working
A practical checklist
References

Pattern 1: Analysis as a substitute for delivery

What it looks like

A team spends months (sometimes a year) doing “analysis” for a capability that won’t be used until it’s built - often with the intention of eliminating all risk up front.

Common examples:

multi-tenant “high availability image storage” designed from scratch
designing bespoke event systems when managed queues exist
writing 40-page architecture documents before the first running slice exists

Why it existed

When provisioning took weeks and environments were scarce, analysis was a rational risk-reducer.

The hidden tax

You push real learning to the end (integration failures happen late).
Decisions get made with imaginary constraints, not measured ones.
Teams optimize for “approval” rather than “outcome.”

The replacement pattern

Timebox discovery and require a running slice early.

A strong default:

1-2 week spike to validate constraints
a thin vertical slice in production (even behind a flag)
iterate based on real telemetry and user feedback

Transition step (low drama)

Create an “RFC-lite” template:

problem statement + constraints
1-2 options with tradeoffs
a plan to measure (latency, cost, reliability)
a thin-slice milestone date

Pattern 2: Reinventing commodity infrastructure

What it looks like

Teams treat widely-proven primitives as novel:

object storage
queues
identity
metrics + tracing
load balancing

A classic symptom: “We need to design HA multi-tenant object storage,” as if durable object storage isn’t already a standard building block.

Why it existed

On-prem and early hosting eras forced you to build a lot yourself.

The hidden tax

Reinventing primitives becomes a multi-quarter project.
Reliability becomes your problem (and you will be on call for it).
The business pays for the same capability twice: once in time, and again in incidents.

The replacement pattern

Default to managed or proven primitives unless you have a documented reason not to.

For example, modern object storage services are explicitly designed for very high durability and availability (provider details vary). [11]

Transition step

Maintain a “Reference Implementations” catalog:

“How we do object storage”
“How we do queues”
“How we do auth”
“How we do telemetry”

If the default is documented and supported, teams stop re-litigating fundamentals.

Pattern 3: VM-first thinking as the default

What it looks like

Everything runs on VMs because “that’s what we do,” even when the workload is a stateless API, worker, or event consumer.

Why it existed

VMs were the universal unit of deployment for a long time, and they map cleanly to org boundaries (“this server is mine”).

The hidden tax

drift (snowflake servers)
slow rollouts
inconsistent security posture
wasted compute due to poor bin-packing
limited standardization across services

The replacement pattern

For many enterprise services, containers orchestrated by Kubernetes are a strong default for stateless workloads. Kubernetes itself describes Deployments as a good fit for managing stateless applications where Pods are interchangeable and replaceable. [5]

This doesn’t mean “Kubernetes for everything,” but it does mean:

prefer declarative workloads with health checks and rollout controls
keep VMs for deliberate cases (legacy constraints, special licensing, unique state, or when orchestration adds no value)

Transition step

Start with “Kubernetes-first for new stateless services,” not a migration mandate.

Then build operational guardrails:

resource requests/limits so services behave predictably under load [6]
standardized readiness/liveness probes
standard ingress + auth patterns

Pattern 4: Ticket-driven infrastructure

What it looks like

Need a database? Ticket. Need an environment? Ticket. Need DNS? Ticket. Need a queue? Ticket.

Eventually, the ticketing system becomes the true control plane.

Why it existed

It’s a reasonable response when:

environments are scarce
changes are risky
platform knowledge is specialized

The hidden tax

queues become normalized (“it takes 3 weeks to get a namespace”)
teams route around the platform
reliability doesn’t improve; delivery just slows

The replacement pattern

Self-service via GitOps and platform “paved roads.”

OpenGitOps describes GitOps as a set of standards/best practices for adopting a structured approach to GitOps. [7] The point isn’t a specific tool - it’s the principle: desired state is declarative and auditable.

Transition step

Pick one high-frequency request and eliminate it:

“create a service with a standard ingress/auth/telemetry”
“provision a queue”
“create a dev environment”

Make the paved road the path of least resistance.

Pattern 5: Change Advisory Board for routine changes

What it looks like

Every change - routine or risky - requires synchronous approval.

Why it existed

When changes were large, rare, and manual, centralized review reduced catastrophic surprises.

The hidden tax

you batch changes (bigger releases are riskier)
emergency changes bypass process (creating inconsistency)
“approval” becomes the goal rather than evidence of safety

DORA’s guidance on streamlining change approval emphasizes making the regular change process fast and reliable enough that it can handle emergencies, and reframes how CAB fits into continuous delivery. [3] Continuous delivery literature makes a similar point: smaller, more frequent changes reduce risk and ease remediation. [4]

The replacement pattern

Move to evidence-based change approval:

automated tests
policy-as-code checks
progressive delivery (canaries, phased rollouts)
real-time telemetry tied to the release

Transition step

Keep CAB, but change its scope:

focus on high-risk changes and cross-team coordination
use automation and metrics for routine changes

Pattern 6: The shared database empire

What it looks like

A central database is shared by many services. Teams coordinate schema changes across multiple apps and releases.

Microservices.io describes the “shared database” pattern explicitly: multiple services access a single database directly. [10]

Why it existed

It’s simple at first:

one place for data
easy joins
one backup plan

The hidden tax

coupling spreads everywhere
every change becomes cross-team work
reliability suffers because one DB problem becomes everyone’s problem
schema evolution becomes political

The replacement pattern

Prefer service-owned data boundaries. Microservices.io’s “database per service” pattern describes keeping a service’s data private and accessible only via its API. [9]

Transition step

You don’t have to “microservices everything.” Start by:

carving out new tables owned by one service
introducing an API boundary
migrating consumers gradually

Pattern 7: Central integration as a chokepoint

What it looks like

All integrations must go through a single shared integration layer/team (classic ESB gravity).

Why it existed

Centralizing integration gave consistency when:

protocols were messy
tooling was expensive
teams lacked automation

The hidden tax

integration lead times explode
teams stop experimenting
one backlog becomes everyone’s bottleneck

The replacement pattern

Standardize:

interfaces (auth, tracing, deployment, contract testing)
platform guardrails

…not every internal implementation detail.

Transition step

Carve out one “self-service integration” paved road:

standard service template
standard auth
standard telemetry
contracts + examples

Pattern 8: Perma-POCs and innovation theater

What it looks like

Prototypes exist forever, never becoming production systems.

Especially common with AI initiatives:

impressive demos
no production constraints
no ownership for operability

Why it existed

POCs are a safe way to explore unknowns.

The hidden tax

teams lose trust (“innovation never ships”)
production teams inherit half-baked work
opportunity cost compounds

The replacement pattern

From day one, require:

an owner
a production path
a thin slice in a real environment
explicit safety requirements (timeouts, budgets, telemetry)

Transition step

Make “POC exit criteria” mandatory:

what metrics prove value?
what is the minimum shippable slice?
what must be true for reliability and security?

Replace committees with guardrails

A recurring theme: humans are expensive control planes.

The modern move is to convert “tribal rules” into:

templates
automation
policy-as-code
paved paths

Microsoft’s platform engineering work describes “paved paths” within an internal developer platform as recommended paths to production that guide developers through requirements without sacrificing velocity. [8]

Guardrails beat gatekeepers because guardrails are:

consistent
fast
auditable
scalable

Modernize without a rewrite

Big-bang rewrites are expensive and risky. Incremental modernization is usually the winning move.

The Strangler Fig pattern is a well-known approach: wrap or route traffic so you can replace parts of a legacy system gradually. [12]

Practical approach:

put a facade in front of the legacy surface
carve off one slice at a time
measure outcomes
keep rollback easy

This isn’t glamorous. It works.

Verification: how you know it’s working

If you want to avoid “modernization theater,” measure.

DORA’s metrics guidance is a solid baseline: deployment frequency, lead time for changes, change failure rate, and time to restore service (MTTR). [1] The 2024 DORA report continues to focus on the organizational capabilities that drive high performance. [2]

A simple evidence loop:

Pick one value stream (one product or platform slice).
Baseline the four DORA metrics.
Remove one friction point (one pattern).
Re-measure.

If your metrics don’t move, you didn’t remove the real constraint.

A practical checklist

If you’re trying to retire “enterprise debt” safely:

Delivery

Timebox analysis; require a running slice early.
Prefer small changes and frequent releases; avoid batching.

Platform

Provide a paved road for common workflows (service template, auth, telemetry). [8]
Remove ticket queues for repeatable requests (self-service + GitOps). [7]

Reliability

Standardize timeouts, retries, budgets, and resource requests/limits. [6]
Use progressive delivery where risk is high.

Architecture

Reduce shared DB coupling; establish service-owned boundaries. [9][10]
Modernize incrementally (Strangler Fig), not via big-bang rewrites. [12]

Governance

Replace routine approvals with evidence: tests + policy-as-code + telemetry. [3][4]

References

[1] DORA - “DORA’s software delivery performance metrics (guide)”. https://dora.dev/guides/dora-metrics/ [2] DORA - “Accelerate State of DevOps Report 2024”. https://dora.dev/research/2024/dora-report/ [3] DORA - “Streamlining change approval (capability)”. https://dora.dev/capabilities/streamlining-change-approval/ [4] ContinuousDelivery.com - “Continuous Delivery and ITIL: Change Management”. https://continuousdelivery.com/2010/11/continuous-delivery-and-itil-change-management/ [5] Kubernetes docs - “Workloads (Deployments are a good fit for stateless workloads)”. https://kubernetes.io/docs/concepts/workloads/ [6] Kubernetes docs - “Resource Management for Pods and Containers (requests/limits)”. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ [7] OpenGitOps - “What is OpenGitOps?” and project background. https://opengitops.dev/ and https://opengitops.dev/about/ [8] Microsoft Engineering Blog - “Building paved paths: the journey to platform engineering”. https://devblogs.microsoft.com/engineering-at-microsoft/building-paved-paths-the-journey-to-platform-engineering/ [9] Microservices.io - “Database per service” pattern. https://microservices.io/patterns/data/database-per-service [10] Microservices.io - “Shared database” pattern. https://microservices.io/patterns/data/shared-database.html [11] AWS documentation - “Data protection in Amazon S3 (durability/availability design goals)”. https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html [12] Martin Fowler - “Strangler Fig Application” (legacy modernization pattern). https://martinfowler.com/bliki/StranglerFigApplication.html

Chapter 13: Templates + Checklists: The Copy/Paste Kit

Sat, 07 Feb 2026 04:00:00 -0500

Series: LLM Development Guide

Chapter 13 of 16

Previous: Chapter 12: Team Collaboration: Handoffs, Shared Prompts, and Review

Next: Chapter 14: Building a Prompt Library: Governance + Quality Bar

What you’ll be able to do

You’ll be able to bootstrap the workflow in minutes:

Create plan/, prompts/, and work-notes/ with consistent templates.
Add a phase spec template for large, multi-phase projects.
Add a phase implementation prompt template for prompt-by-prompt execution.
Use session start and end checklists.
Generate PR descriptions that explain intent and verification.

TL;DR

Templates reduce prompt drift.
Keep them short and consistent.
Add verification to every phase.
For large projects, pair phase specs with implementation prompt docs.

Plan template
Prompt template
Phase spec template (large projects)
Phase implementation prompt template (large projects)
Work notes template
Session checklists
PR description template
Verification

Plan template

# <Project> Plan

## Overview

## Goals
-

## Context
- Reference implementation:
- Environment:

## Phases

## Definition of done
- [ ]

## Out of scope
-

## Risks / open questions
-

Prompt template

# Phase <X> - <Name>

## Role

## Context
- Plan:
- Work notes:
- References:

## Task

## Deliverables
1.

## Constraints
- MUST
- MUST NOT

## Session management
Update work notes with decisions, assumptions, open questions, and a session log entry.

## Verification
- Command:
- Expected:

## Commit discipline
Propose a commit message and wait for approval.

Phase spec template (large projects)

# Phase <N><Letter> - <Phase Name>

## Status
Planned

## Depends on
- <Phase dependency>

## Feature flag
- <flag name or n/a>

## Migration
- <none or required steps>

## Design rationale
<Why this phase exists and what risk it reduces>

## Tasks
### Prompt 1
- <task>

### Prompt 2
- <task>

## Files
### New
- <path>

### Modified
- <path>

### Referenced (read-only)
- <path>

## Exit criteria
- [ ] <build command> exits 0
- [ ] <vet/lint command> exits 0
- [ ] <test command> exits 0
- [ ] No ignored returned errors

## Progress notes

Phase implementation prompt template (large projects)

# Phase <N><Letter> - Implementation Prompts

Complete prompts sequentially. Do not continue when verification fails.

## Prompt 1 of <Total>: <Prompt Name>

Context files to load:
- <4 to 6 explicit paths>

Task:
- <exact implementation task>

Constraints:
- Stay within this prompt's scope.
- Handle all returned errors.
- Keep code small and reviewable.
- Do not proceed to the next prompt until verification passes.

Verification:
- <build command>
- <vet/lint command>
- <test command>

Commit discipline:
- Summarize what changed.
- Propose commit message.
- Wait for approval before moving on.

Work notes template

# Phase <X> - <Name>

## Status
- [ ] Not started
- [ ] In progress
- [ ] Blocked
- [ ] Complete

## Decisions

## Assumptions

## Open questions

## Session log

## Commits

Session checklists

Session start:

Identify the phase.
Load the phase prompt.
Load the current work notes.
Re-state the smallest goal for this session.

Session end:

Work notes updated.
Decisions logged with rationale.
Verification run.
Commits made (or clearly blocked).
Next step written down.

PR description template

## Summary

## Changes
-

## Out of scope
-

## Verification
-

## Review guide

## Follow-up
- [ ]

## References
- Work notes:
- Plan:

Verification

Create a local templates/ folder and seed the files:

mkdir -p templates

cat > templates/PLAN-template.md <<'MD'
# Project Plan

## Overview

## Goals

## Phases

## Definition of done
MD

cat > templates/PROMPT-template.md <<'MD'
# Phase - Prompt

## Role

## Context

## Task

## Constraints

## Verification

## Commit discipline
MD

cat > templates/WORK-NOTES-template.md <<'MD'
# Phase - Work Notes

## Status

## Decisions

## Session log
MD

Expected result:

You can start a new project by copying these templates and editing the placeholders.

Continue -> Chapter 14: Building a Prompt Library: Governance + Quality Bar

Chapter 12: Team Collaboration: Handoffs, Shared Prompts, and Review

Thu, 05 Feb 2026 02:00:00 -0500

Series: LLM Development Guide

Chapter 12 of 16

Previous: Chapter 11: Measuring Success: Solo + Team Metrics Without Fake Precision

Next: Chapter 13: Templates + Checklists: The Copy/Paste Kit

What you’ll be able to do

You’ll be able to run this workflow on a team without turning it into process theater:

Hand off work mid-phase without a meeting.
Share prompts that actually work.
Review LLM-assisted code with the same rigor as human code.

TL;DR

Teams fail at LLM work because chat context is not shareable.
Plans, prompt docs, and work notes make context portable.
Keep review focused on code and verification, not on how the code was produced.
Maintain a small set of “golden” reference implementations.

Handoff patterns
Shared prompt libraries
Review checklist
Verification

Handoff patterns

Mid-phase handoff

If you hand off in the middle of a phase, provide:

Updated work notes with status, decisions, open questions, and exact next step.
The phase prompt doc.
The reference implementation paths used.
Any verification output (test results, lint output).

Handoff template:

## Handoff: <Phase>

### Status
<What's done, what's in progress, what's blocked>

### Files to review
- <file 1>
- <file 2>

### Key decisions
- <Decision>: <Rationale>

### Open questions
- [ ] <Question>

### Immediate next step
<Exact command or file edit to do next>

### How to resume
1. Load prompts/<phase>.md
2. Load work-notes/<phase>.md
3. Continue from the last session log entry

Phase boundary handoff

Phase boundary handoffs are easier:

Work notes are marked complete.
The next phase starts cleanly.

Shared prompt libraries

A shared library reduces rework and increases consistency.

A reasonable structure:

prompt-library/
 planning/
 implementation/
 testing/
 review/

Quality bar:

Prompts are specific enough to be useful.
Prompts are general enough to be reused.
Prompts record “when to use” and “prereqs”.
Prompts have been used successfully multiple times.

Review checklist

LLM-assisted work should be reviewed like any other work.

High-signal checks:

Imports and APIs exist (no hallucinations).
Error handling is complete.
Output matches reference patterns.
Verification was actually run.
Commits are atomic and explain intent.
Tests test behavior, not just existence.

Verification

Create a shared template file so handoffs are consistent:

mkdir -p docs

cat > docs/llm-handoff-template.md <<'MD'
# LLM Work Handoff Template

## Phase

## Status

## Files to review

## Key decisions

## Open questions

## Verification run
- <command>
- Expected: <...>

## Next step

## How to resume
- Prompt:
- Work notes:
- References:
MD

Expected result:

Anyone can hand off work in under five minutes.

Continue -> Chapter 13: Templates + Checklists: The Copy/Paste Kit

Chapter 11: Measuring Success: Solo + Team Metrics Without Fake Precision

Tue, 03 Feb 2026 00:00:00 -0500

Series: LLM Development Guide

Chapter 11 of 16

Previous: Chapter 10: Stop Rules + Pitfalls: When to Upgrade, Bail, or Go Manual

Next: Chapter 12: Team Collaboration: Handoffs, Shared Prompts, and Review

What you’ll be able to do

You’ll be able to tell, with reasonable honesty, whether the workflow is helping:

Pick a small set of metrics you can actually measure.
Separate leading indicators (process) from lagging indicators (outcomes).
Avoid fake precision and vanity metrics.

TL;DR

If you can’t measure reliably, don’t invent numbers.
Track a baseline (a few representative tasks) before you claim improvement.
Favor cheap metrics: time to first commit, PR revision rounds, post-merge bugs.
Use leading indicators daily; use lagging indicators in retros.

What to measure
Solo baseline
Leading vs lagging indicators
Lightweight reporting template
Verification

What to measure

Pick a small set that maps to real outcomes.

Velocity indicators:

Time to first commit.
Phase completion time.
PR cycle time.

Quality indicators:

PR revision rounds.
Bugs caught in review.
Post-merge bugs.

Efficiency indicators:

Rework rate (time fixing output vs total time).
Session count per task.
Handoff success (can someone else continue without re-explaining).

Solo baseline

If you’re working solo, you can still create a baseline.

Track per task:

Start time.
First commit time.
Total time to done.
Number of “LLM retries” (how many prompt iterations for the same logical unit).
Bugs you found after “done”.

The point is not perfect measurement. The point is noticing patterns.

Leading vs lagging indicators

Leading indicators predict success:

Work notes are updated.
Prompts contain verification.
Commits are atomic.
References are provided.

Lagging indicators confirm success:

PR merged with low rework.
Low post-merge bug rate.
Handoffs succeed.

Lightweight reporting template

## LLM-Assisted Development Summary (Month)

### Adoption
- Tasks completed with workflow: <N>

### Velocity
- Median time to first commit: <X>
- Median PR cycle time: <Y>

### Quality
- Median PR revision rounds: <Z>
- Post-merge bugs: <N>

### Costs
- LLM cost estimate: <X>

### Notes
- What worked:
- What failed:
- Changes for next month:

Verification

Keep a simple CSV so you can graph later if you want.

mkdir -p work-notes

cat > work-notes/metrics.csv <<'CSV'
date,task,time_to_first_commit_minutes,total_time_minutes,llm_retries,pr_revision_rounds,post_merge_bugs,notes
CSV

Expected result:

You can append one row per task in under a minute.

Continue -> Chapter 12: Team Collaboration: Handoffs, Shared Prompts, and Review

Go vs Spring Boot for Enterprise APIs: Cost, Performance, and Cloud-Native Ops

Sun, 01 Feb 2026 10:00:00 -0500

As-of note: This is a production engineering perspective, not a benchmark scoreboard. If you care about cost or p99 latency, measure your service with your dependencies and your deployment constraints.

Why this comparison keeps showing up

If you build enterprise APIs long enough, you’ll see the same pattern:

The “language choice” isn’t what breaks production.
The runtime envelope and operational model usually are.

When teams compare Go and Java Spring Boot, they’re often asking a more specific question:

“What will it cost to run this API at scale, and how predictable is it under real production conditions?”

Spring Boot’s value proposition is speed-to-service: stand-alone, production-grade Spring applications you can “just run,” with strong ecosystem defaults and integration breadth. [1]

Go’s value proposition is operational simplicity: compile to an executable, ship a small container, run with fewer moving pieces, and keep latency and resource usage easier to reason about. go build compiles packages into an executable. [5]

This article is about the production-relevant tradeoffs: cost/resource usage, performance under load, cloud-native deployability, and the “you will be on call for this” realities.

On code quality: This isn’t “Go good / Java bad.” It’s an observation about failure modes: framework-heavy stacks can hide complexity until it shows up in startup time, memory, and surprises under load. Go’s bias toward explicitness often makes problems easier to see and cheaper to operate, even before the codebase is perfect.

TL;DR

If your org is already Spring-heavy, Spring Boot can be the fastest path to a robust API, especially when you need Spring’s ecosystem (security, data, integrations). [1]
If you run many small services, care about density, or need fast scale-to-zero/scale-from-zero behavior, Go often has an operational edge due to simpler packaging and typically lower baseline resource footprint.
Kubernetes costs are strongly influenced by requests/limits and scheduling density, so baseline memory is often a bigger lever than micro-optimizing CPU. [7][8]
Both ecosystems support hardened container builds (including distroless) to reduce attack surface. [9][10]
Observability is excellent in both; Java has very mature zero-code instrumentation via the OpenTelemetry Java agent. [13][14] Go has strong SDK support and growing options for auto-instrumentation. [11]
“Best” depends on your constraints. The best move is to benchmark your service envelope and compare p95/p99 latency, RSS, startup, and error rates under load.

The cost model: what you actually pay for

In cloud and Kubernetes environments, cost is strongly driven by:

How many replicas you need
How much CPU/memory you request per replica
How quickly you can scale (up and down)
How much time you spend operating the service

Kubernetes scheduling and resource guarantees are based on requests and limits. Requests influence where Pods can be scheduled; limits cap what they can consume. [7][8]

That means your “baseline footprint” matters:

A service that requests 512Mi RAM even when idle reduces node density.
A service that requests 128Mi RAM allows more Pods per node.

A simple (illustrative) density example

Assume you run 100 replicas of an API, and memory is your limiting resource:

Case A: 100 × 512Mi = 51,200Mi ≈ 50Gi reserved
Case B: 100 × 128Mi = 12,800Mi ≈ 12.5Gi reserved

That’s a ~37.5Gi delta in reserved memory before you count overhead (sidecars, DaemonSets, kube-system). This is not “Go vs Java math.” It’s “baseline footprint sets cluster size.”

The point: cost discussions are often memory-and-startup discussions wearing a language-comparison mask.

Go’s production advantages (when they matter)

1) Packaging simplicity and deployment surface

Go’s toolchain compiles code into an executable (go build). [5] Go’s modern toolchain approach (including toolchain selection starting in recent Go releases) helps keep builds reproducible across environments. [6]

In practice, Go services often ship as:

a single process
a single container layer containing a single binary
minimal runtime dependencies

That tends to reduce:

container image complexity
“works on my machine” drift
runtime patch surface area

This matters most when you operate many services and want upgrades to be boring.

2) Fast start and “scale events”

In real systems, performance isn’t only request/response speed, it’s also how the service behaves during:

deployments
autoscaling
node drains
crashes

Go services commonly start quickly because they don’t require JVM warmup/classloading/JIT compilation. (Exact numbers vary; measure your service.)

Spring Boot can start fast enough for most use cases, but cold starts can become a visible factor when:

you scale from zero frequently (serverless-like patterns)
you do aggressive HPA scaling
you run lots of short-lived jobs

Spring Boot also supports building native images with GraalVM, which can materially improve startup and memory in some cases, but introduces different tradeoffs (build time, reflection limits, operational differences). [3][4]

3) Resource envelope predictability

For many “API gateway / orchestration / integration” services, CPU isn’t the bottleneck. Latency, network, and downstream behavior are.

Go’s strengths here tend to be:

predictable concurrency behavior
straightforward backpressure patterns (bounded queues, semaphores)
fewer runtime tuning knobs compared to JVM-heavy stacks

This is not “Go always uses less RAM.” It’s “Go often gives you a tighter baseline envelope for simpler services, which improves scheduling density.”

4) Cloud-native ergonomics: minimalism wins over time

Enterprise services accrete complexity over years. The less your runtime depends on:

classpath complexity
reflection-driven magic
extensive framework graphs

…the easier it is to keep production surprises rare.

Go’s bias toward explicit wiring tends to help with long-term operability, especially in platform/API layers where consistency matters.

Where Spring Boot is still the right tool

Spring Boot exists for a reason, and in many enterprises it’s still the correct default:

1) Ecosystem and “starter” leverage

Spring Boot’s opinionated defaults and starter ecosystem are an enormous accelerator for:

auth (OAuth2/OIDC)
data access and ORM patterns
enterprise integrations
standardized configuration and profiles

Spring Boot is explicitly designed to minimize configuration and help you ship “production-grade” applications quickly. [1]

If you already have:

shared Spring libraries
internal Spring starters
company-wide Spring conventions

…then choosing Go for “purity” can be expensive in human terms.

2) JVM performance can be excellent

For long-lived services under sustained load, HotSpot JIT compilation can deliver extremely strong performance, sometimes outperforming Go in CPU-bound or allocation-sensitive scenarios.

It’s a mistake to assume “compiled native binary” automatically means “faster.” The real question is: p99 latency, throughput per core, and behavior under GC pressure for your workload.

3) Operational maturity and tooling

Spring Boot has well-worn operational patterns:

actuator endpoints
consistent configuration patterns
deep tracing/profiling options
broad community knowledge

Also: if your org has deep Java on-call expertise, “operational simplicity” may already be solved socially.

Cloud-native reality: images, CVEs, and deploy surface

Distroless is not a Go-only advantage

A common Go pattern is “static binary + scratch/distroless.” But distroless images exist for Java too.

Distroless images contain only the application and its runtime dependencies, with no package manager and no shell, reducing attack surface. [9] The distroless project includes Java images as well. [10]

Operational implication: smaller, simpler images usually mean:

faster pulls and rollouts
fewer things to patch
fewer “shell inside container” habits (a feature, not a bug)

Whether you ship Go or Spring Boot, you can adopt hardened bases.

Two Dockerfile patterns (illustrative)

Go (multi-stage + distroless):

FROM golang:1.22-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags "-s -w" -o /out/api ./cmd/api

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/api /api
USER nonroot:nonroot
ENTRYPOINT ["/api"]

Spring Boot (JAR + distroless Java):

FROM eclipse-temurin:21-jdk AS build
WORKDIR /src
COPY . .
RUN ./mvnw -DskipTests package

FROM gcr.io/distroless/java21-debian12:nonroot
COPY --from=build /src/target/app.jar /app.jar
USER nonroot:nonroot
ENTRYPOINT ["java","-jar","/app.jar"]

The important part isn’t the exact base image, it’s the principle: reduce image surface area and keep the deploy artifact boring.

Observability and operations

Both ecosystems are strong here, but they differ in “how quickly can I get real telemetry.”

OpenTelemetry support

OpenTelemetry is the vendor-neutral standard for traces/metrics/logs. [11]

Go language docs: SDK + instrumentation guidance. [11]
Java language docs: SDK + instrumentation guidance. [12]

Java’s advantage: zero-code instrumentation

The OpenTelemetry Java agent can attach to Java applications and automatically instrument popular libraries via bytecode injection. [13] The OpenTelemetry Java instrumentation project provides the agent and broad library coverage. [14]

Practical implication: you can often get useful traces without touching code. That’s a meaningful ops advantage in large enterprises.

Go’s reality: explicit instrumentation (plus growing options)

Go’s OpenTelemetry SDK support is strong. [11] Go auto-instrumentation options exist and are improving, but your fastest path today is still typically:

instrument key inbound/outbound edges in code
standardize middleware across services
treat telemetry as part of the API contract

That’s not bad. It’s just a different default.

A decision matrix

Use this as a starting point, not a rule.

Constraint / Goal	Go tends to win	Spring Boot tends to win
Many small services, high density	✅ smaller baseline envelopes often help	⚠️ can be heavier per-service
Fast scale-from-zero, frequent redeploys	✅ typically quick startup	✅ with care; ✅✅ with native image tradeoffs [3][4]
Enterprise integration breadth	⚠️ you build more glue yourself	✅ Spring ecosystem leverage [1]
Team expertise	✅ if Go is your platform standard	✅ if Java/Spring is your standard
“Boring deployments”	✅ single binary patterns	✅ well-trodden JVM patterns
Zero-code observability	⚠️ emerging	✅ OTel Java agent maturity [13][14]
Long-lived CPU-heavy services	✅ sometimes	✅ JVM can be extremely strong

How to validate with a real experiment

If you want a decision you can defend, run a 2-4 hour experiment:

1) Define a representative endpoint mix

1 simple “health/read” endpoint
1 endpoint that hits your DB
1 endpoint that calls a downstream HTTP service
1 endpoint with payload validation + auth

2) Measure the four numbers that matter

Startup time (cold start to ready)
Steady-state RSS at idle
p95 / p99 latency under load
Error rate under load + partial downstream failure

3) Run the same load and failure profile

Use the same:

container runtime
resource requests/limits
ingress configuration
downstream simulators

4) Compare operational work, not only performance

How painful is debugging?
How much config is required?
How quickly can your team ship fixes safely?

This is where enterprise reality lives.

Common failure modes

Go pitfalls

Teams reinvent frameworks inconsistently across services.
Too much “just a handler” code without shared middleware for auth, limits, tracing, and error handling.
Ignoring backpressure (unbounded goroutines) → memory blowups.

Spring Boot pitfalls

Default dependency graphs grow quietly until startup time and memory become a problem.
Classpath/auto-config complexity makes “why did it do that?” debugging expensive.
Container runtime tuning gets deferred, then becomes urgent during cost reviews.

Both ecosystems

No explicit timeouts (inbound and outbound).
No limits or budgets.
No telemetry until after the first incident.

Closing thought

If your enterprise APIs are:

small, numerous, latency-sensitive, and cost-sensitive
…Go is often a strong default.

If your enterprise APIs are:

integration-heavy, domain-rich, and built on existing Spring conventions
…Spring Boot is usually the shortest path to “production-grade.”

The best answer is the one you can operate confidently, on call, at scale.

References

Spring Boot project overview:
Spring Boot reference: Graceful Shutdown:
Spring Boot reference: GraalVM Native Images:
GraalVM guide: Build a Spring Boot app into a native executable:
Go tutorial: Compile and install the application (go build produces an executable):
Go docs: Toolchains and the go command:
Kubernetes docs: Resource Management for Pods and Containers (requests/limits):
Google Cloud: Kubernetes best practices for resource requests and limits:
Distroless container images (project overview):
Distroless Java images:
OpenTelemetry Go docs:
OpenTelemetry Java docs:
OpenTelemetry Java Agent (zero-code):
OpenTelemetry Java instrumentation (agent JAR + library coverage):

Chapter 10: Stop Rules + Pitfalls: When to Upgrade, Bail, or Go Manual

Sat, 31 Jan 2026 23:00:00 -0500

Series: LLM Development Guide

Chapter 10 of 16

Previous: Chapter 9: Security & Sensitive Data: Sanitize, Don’t Paste Secrets

Next: Chapter 11: Measuring Success: Solo + Team Metrics Without Fake Precision

What you’ll be able to do

You’ll be able to avoid the two common failure outcomes:

Spending hours fighting the model.
Shipping output you can’t review.

You’ll do it with explicit stop rules, upgrade triggers, and a short recovery checklist.

TL;DR

If the change is under a minute manually, do it manually.
If you can’t review the output competently, don’t ship it.
If you’re on your third attempt for the same logical unit, upgrade or re-scope.
Add verification steps to plans and prompts so “done” is testable.

Stop rules
Top pitfalls
Recovery checklist
Verification

Stop rules

These are pragmatic defaults. Tune them to your environment.

Stop rule 1: tiny changes

If it is a tiny change (one line, one rename, one version bump), do it manually.

LLM overhead is real:

You still have to explain.
You still have to review.
You still have to verify.

Stop rule 2: you can’t review it

Never commit code you could not explain in a review.

If you don’t understand the domain:

break the work into smaller pieces you can understand, or
involve a reviewer who does.

Stop rule 3: you’re fighting output quality

The 10-minute rule:

If you’ve spent about 10 minutes fighting the output, stop.
Upgrade the model tier, or shrink the scope to a smaller logical unit.

Stop rule 4: high-risk code needs extra caution

Be cautious with:

Authentication and authorization.
Cryptography.
Payment flows.
Input validation.

You can still use LLMs, but the bar for review and verification is higher.

Top pitfalls

These show up repeatedly.

Trusting output without review.
Skipping planning.
Not providing reference implementations.
Letting sessions run too long.
Scope creep mid-session.
Vague prompts.
Not capturing decisions.
No verification step.

A simple rule:

If you wouldn’t merge a junior developer’s PR without review, don’t merge LLM output without review.

Recovery checklist

When things go wrong:

Stop iterating on bad output.
Decide what kind of problem it is:
- prompt problem,
- model capability problem,
- task is a poor fit.
Simplify:
- smaller logical unit,
- more references,
- clearer constraints.
Fresh session if context has drifted.
Manual fallback is a valid outcome.

Verification

Create a one-page stop-rules file so you can apply this consistently across tasks:

mkdir -p work-notes

cat > work-notes/stop-rules.md <<'MD'
# Stop Rules (Personal Defaults)

## Manual first
- If change is <= 1 minute manually, do it manually.

## Upgrade triggers
- Third attempt on same logical unit.
- Repeated misunderstandings.
- Output ignores constraints.

## Bail triggers
- I cannot review this competently.
- Task requires live debugging with runtime state.
- Sensitive data would be required to reproduce.

## Required gates
- Verification commands exist in plan.
- Verification commands exist in prompt.
- Work notes updated before continuing.
MD

Expected result:

You have a written policy you can apply without debating every time.

Continue -> Chapter 11: Measuring Success: Solo + Team Metrics Without Fake Precision

Stop Shipping Slide Decks

Sat, 31 Jan 2026 11:15:00 -0500

Position: This is not “documentation bad.” This is “documentation is a tool.” If it increases lead time, hides truth, or replaces learning, it’s not helping.

Why this matters

In software, the real “source of truth” is:

running systems
code and configuration
production telemetry
incident history

Documentation should reduce uncertainty and speed up decisions. But two artifacts routinely do the opposite in large organizations:

the 40-page slide deck
the Word doc living somewhere in SharePoint that nobody can find

These artifacts often become deliverables - a substitute for building. They make it possible to spend months “progressing” without ever encountering reality.

And here’s the part most orgs miss:

If you’re going to fail, you want to fail quickly and cheaply, not slowly and expensively. [4]

That doesn’t mean reckless shipping. It means running a tight learning loop and letting reality correct you early - before you’ve sunk quarters of time into the wrong solution.

TL;DR

Decks are great for storytelling. They are bad as an engineering system of record.
“SharePoint architecture docs” become a document cemetery: hard to find, hard to diff, and easy to ignore.
The Agile Manifesto explicitly values working software over comprehensive documentation. [1] And one Agile principle states that working software is the primary measure of progress. [2]
Replace decks/docs-as-deliverables with:
RFC-lite (1-2 pages) + a running thin slice
ADRs (Architecture Decision Records) to capture decisions + tradeoffs [5][6]
Docs-as-code (Markdown in the repo, reviewed like code)
diagrams that are versioned and easy to update
Measure improvement with system outcomes (lead time, deploy frequency, change failure rate, MTTR). [3]

Pattern 1: Deck-driven development
Pattern 2: SharePoint document cemeteries
Pattern 3: Architecture as narrative, not decisions
Pattern 4: “Design phase” gating
Pattern 5: Documentation that never gets pruned
What to do instead: a documentation system that ships
Verification: how you know it’s working
A practical checklist
References

Pattern 1: Deck-driven development

What it looks like

A 40-page deck is created to describe a system that doesn’t exist yet.
The deck gets reviewed by multiple groups.
Approval is treated as progress.
When implementation starts, the world has changed - or key constraints were missed.

Why it exists

Decks are socially useful:

they compress complexity into a narrative
they help leaders “see” a plan
they make uncertainty feel controlled

The hidden tax

Decks are a poor engineering artifact because they’re:

low fidelity: they rarely contain executable truth
hard to maintain: updates are manual and usually lag reality
hard to diff: you can’t easily review what changed and why
easy to perform: a deck can look complete while the design is still untested
not tied to code: no direct path from “decision” -> “implementation” -> “verification”

The worst outcome isn’t that the deck is wrong. It’s that the deck delays the point where you discover what’s wrong.

The replacement pattern

Use decks for storytelling after you have reality. Use engineering artifacts to discover reality.

A strong default:

RFC-lite (1-2 pages)
a runnable thin slice
measurable verification (latency, cost envelope, failure mode)

This aligns with Agile’s emphasis on working software as a real measure of progress. [2]

Transition step (low drama)

Replace “deck required for approval” with “evidence required for approval”:

link to the RFC
link to a running demo / branch / sandbox
explicit constraints + tradeoffs
an exit criteria checklist for the slice

Pattern 2: SharePoint document cemeteries

What it looks like

Architecture docs exist as Word/PDF files in SharePoint.
Multiple versions exist (“Final_v7_REAL_FINAL.docx”).
Search works poorly unless you already know what to search for.
Nobody updates the doc because it’s painful and risky (“what if I change the blessed doc?”).

Why it exists

It’s an enterprise default:

SharePoint is “official”
Word docs feel formal
it’s familiar to non-engineering stakeholders

The hidden tax

SharePoint docs typically fail at the things engineering needs most:

discoverability (people don’t know where to look)
ownership (no clear maintainer)
reviewability (diffs and PR discussion are weak)
linking to reality (code, configs, dashboards, runbooks)
keeping current (documentation drift becomes the norm)

So teams stop trusting docs and rely on tribal knowledge - until they page someone at 2 a.m.

The replacement pattern

Treat documentation as part of the codebase:

Markdown in the repo
reviewed via PR like code
versioned with implementation
linked to:
APIs (OpenAPI specs)
dashboards
runbooks
incident writeups
ADRs

Google’s documentation best practices make the point directly: a small set of fresh, accurate docs is better than a large pile in disrepair. [7]

Transition step

You don’t have to “migrate all docs.”

Start with a triage:

Identify the top 10 documents people actually need.
Recreate them as Markdown in a docs/ folder with an index.
Leave the rest as archived references, not living truth.

Pattern 3: Architecture as narrative, not decisions

What it looks like

The doc describes a target architecture but doesn’t answer:

why this approach?
what alternatives were considered?
what tradeoffs were accepted?
what constraints matter most?
what did we decide not to do?

Why it exists

Narratives are easier than decision logs. It’s simpler to write “the system will…” than to record the messy reality of tradeoffs.

The hidden tax

When decisions aren’t recorded, teams re-litigate them repeatedly. The same arguments come back every quarter - often because new people joined and the reasoning isn’t captured.

The replacement pattern: ADRs

Use Architecture Decision Records (ADRs): short, structured notes that capture an important decision with its context and consequences. [5] The practice is commonly attributed to Michael Nygard’s 2011 write-up. [6]

ADRs are the opposite of a 40-slide deck:

small
specific
diffable
linkable to code changes

Transition step

Start with one ADR per “architecturally significant decision”:

database choice
messaging pattern
tenancy model
auth model
deployment model
data boundary decisions

Pattern 4: “Design phase” gating

What it looks like

“We can’t start implementation until the analysis is complete.”
The analysis expands to include every possible future case.
The design grows more “complete” and less true.

Why it exists

Enterprises are understandably afraid of failure.

The hidden tax

This approach doesn’t eliminate failure. It defers it - making it more expensive.

Lean Startup describes progress as validated learning and emphasizes moving quickly through a build-measure-learn loop. [4] The point isn’t startups. The point is learning fast when you’re uncertain.

The replacement pattern

Timebox design, then validate with a thin slice:

write the RFC-lite doc
implement the smallest realistic end-to-end path
measure the constraints
then expand

Transition step

Define “analysis exit criteria”:

measurable constraints validated (not theorized)
spike code exists
a plan for incremental rollout exists

Pattern 5: Documentation that never gets pruned

What it looks like

Docs accumulate but aren’t maintained:

outdated architecture diagrams
old runbooks
stale onboarding guides
dead links

Why it exists

Pruning isn’t rewarded. Writing new docs feels productive; deleting old docs feels risky.

The hidden tax

Stale docs are worse than no docs:

they mislead
they increase cognitive load
they create false confidence

The replacement pattern

Adopt “minimum viable documentation” and prune regularly. [7]

The rule I like:

If a doc isn’t maintained, label it ARCHIVED and explain why.
If a doc is required, tie it to ownership and change workflow.

Transition step

Make docs part of PR hygiene:

if the change affects behavior, docs update ships with it
run link checks in CI
keep an index page updated

What to do instead: a documentation system that ships

Here’s a simple “docs system” that works in practice.

A repo structure that scales

/README.md # entry point: what this is + how to run it
/docs/
 index.md # "start here" documentation map
 rfc/
 0001-tenancy-model.md
 0002-storage-approach.md
 adr/
 0001-use-postgres.md
 0002-adopt-opentelemetry.md
 architecture/
 context.md # C4-ish: context + boundaries
 containers.md # top-level services
 deployment.md # runtime & environments
 runbooks/
 oncall.md
 incident-response.md
 api/
 openapi.yaml

Replace 40 slides with two artifacts

RFC-lite (1-2 pages): the “what” and “why”
Thin slice demo: the reality check

RFC-lite template (copy/paste)

# RFC: <title>

## Problem
What are we trying to solve? Who is affected?

## Constraints
Latency, cost, compliance, tenancy, uptime, environments.

## Proposal
What are we building? What does "done" mean?

## Alternatives considered
Option A / B / C with short tradeoffs.

## Risks and mitigations
What could go wrong? How will we contain blast radius?

## Verification
How will we measure success in production?

ADR template (copy/paste)

# ADR-XXXX: <decision>

## Status
Proposed | Accepted | Deprecated

## Context
What drove this decision? What constraints matter?

## Decision
What did we decide?

## Consequences
What do we gain? What do we lose? What changes later?

Verification: how you know it’s working

If you replace decks and doc cemeteries with real engineering artifacts, you should see:

Delivery metrics improve

Track the same system-level outcomes DORA promotes: lead time, deploy frequency, change failure rate, and time to restore service. [3]

Fewer handoffs and fewer “alignment meetings”

If teams can self-serve context from living docs, coordination cost drops.

Faster “first reality”

A simple heuristic:

How long from idea -> first runnable thin slice?

If that number is months, the system is optimized for analysis, not learning.

Docs stay alive

docs updated alongside code
fewer stale “final_v7” files
fewer tribal-knowledge escalations

A practical checklist

If you want to kill deck-driven delivery without starting a culture war:

Stop treating decks as deliverables

Architecture reviews require an RFC + a runnable slice.
Decks are optional; evidence is not.

Fix document discoverability

One docs/index.md that links to the docs that matter.
Make the repo the source of truth for technical docs.

Capture decisions, not fantasies

Add ADRs for major decisions and link them to PRs. [5][6]

Timebox analysis

Set analysis exit criteria.
Optimize for early learning and quick failure when uncertainty is high. [4]

Keep docs small and alive

Prune regularly; archive what’s stale.
Run link checks in CI.
Treat docs like bonsai: maintained and trimmed, not accumulated. [7]

References

[1] Manifesto for Agile Software Development (values; “Working software over comprehensive documentation”). https://agilemanifesto.org/ [2] Principles behind the Agile Manifesto (“Working software is the primary measure of progress”). https://agilemanifesto.org/principles.html [3] DORA - “DORA’s software delivery performance metrics (guide)”. https://dora.dev/guides/dora-metrics/ [4] Lean Startup principles (Build-Measure-Learn; learning quickly; failing fast/cheaply as a concept). https://theleanstartup.com/principles [5] ADR - Architectural Decision Records (what ADRs are). https://adr.github.io/ [6] Michael Nygard - “Documenting Architecture Decisions” (2011; ADR practice origin/popularization). https://www.cognitect.com/blog/2011/11/15/documenting-architecture-decisions [7] Google Documentation Guide - Best practices (“Minimum Viable Documentation”; keep docs short, fresh, and pruned). https://google.github.io/styleguide/docguide/best_practices.html

MCP Servers in Production: Hardening, Backpressure, and Observability (Go)

Sat, 31 Jan 2026 09:00:00 -0500

As-of note: MCP is evolving. This article references the MCP specification versioned 2025-11-25 and related docs; verify details against the current spec before shipping changes. [1][2][4]

Why this matters

Most “agent demos” fail in production for boring reasons: missing timeouts, unbounded concurrency, ambiguous tool interfaces, and logging that accidentally turns into data exfiltration.

An MCP server isn’t “just an integration.” It’s a capability boundary between an LLM host (IDE, desktop app, agent runner) and the real world: files, APIs, databases, tickets, home automation, and anything else you wire up. MCP uses JSON-RPC 2.0 messages over transports like stdio (local) and Streamable HTTP (remote). [1][2][5]

That means an MCP server is:

an API gateway for tools
a policy enforcement point (whether you intended it or not)
a reliability hotspot (tool calls are where latency and failure concentrate)
a security hotspot (tools are where “read” becomes “exfil” and “write” becomes “impact”)

This post is a pragmatic checklist + a set of Go patterns to harden an MCP server so it keeps working when it’s under real load, and remains safe when the model gets “creative.”

TL;DR

Treat tool inputs as untrusted. Validate and constrain everything.
Put budgets everywhere: timeouts, concurrency limits, rate limits, and payload caps.
Build for partial failure: retries, idempotency keys, circuit breaking, fallbacks.
Log like a security engineer: structured, redacted, auditable, and useful. [11]
Instrument with traces/metrics early; “we’ll add telemetry later” is a trap. [13]
Prefer Go for MCP servers because deployment and operational behavior are predictable: single binary, fast startup, structured concurrency via context, and a strong standard library.

A production mental model for MCP servers
Threat model: what actually goes wrong
Hardening layer 1: identity and authorization
Hardening layer 2: tool contracts that resist ambiguity
Hardening layer 3: budgets and backpressure
Hardening layer 4: safe networking and SSRF containment
Hardening layer 5: observability without leaking secrets
Hardening layer 6: versioning and rollout discipline
A production checklist
References

A production mental model for MCP servers

MCP’s docs describe a host (the AI application), a client (connector inside the host), and servers (capabilities/providers). Servers can be “local” (stdio) or “remote” (Streamable HTTP). [2][3]

Here’s the production mental model that matters:

Your MCP server is a tool gateway.
Every tool is effectively an RPC method exposed to an agent. MCP uses JSON-RPC 2.0 semantics for requests/responses/notifications. [1][5]
LLM tool arguments are not trustworthy.
Even if the LLM is “helpful,” arguments can be malformed, overbroad, or dangerous, especially under prompt injection or user-provided hostile input.
The host UI is not a security boundary.
The spec emphasizes user consent and tool safety, but the protocol can’t enforce your policy for you. You still need server-side controls. [1]
Transport changes your blast radius, not your responsibilities.
Stdio reduces network exposure, but doesn’t remove safety requirements. Streamable HTTP adds multi-client/multi-tenant concerns and requires real auth. [2][3]

If you remember nothing else: treat the MCP server like a production API you’d be willing to put on call for.

Threat model: what actually goes wrong

When MCP servers cause incidents, it’s usually one of these:

1) Input ambiguity → destructive actions

A “delete” tool with optional filters
A “run command” tool with free-form strings
A “sync” tool that can touch thousands of objects

Mitigation: schema + semantic validation, safe defaults, two-phase commit patterns (preview then apply), and explicit “danger gates.”

2) Prompt injection → tool misuse

The model can be tricked into calling tools with attacker-provided arguments. If your tool can read internal data or call internal APIs, you’ve created an exfil path.

Mitigation: least privilege, allowlists, strong auth, egress controls, and redaction.

3) SSRF / network pivoting

Any tool that fetches URLs, loads webhooks, or calls dynamic endpoints can be abused to hit internal networks or metadata endpoints. OWASP treats SSRF as a major category for a reason. [10]

Mitigation: deny-by-default networking (CIDR blocks, DNS/IP resolution checks, allowlisted destinations).

4) Unbounded concurrency → resource collapse

Agents can fire tools in parallel. Without limits you’ll blow up:

API quotas
DB connections
CPU/memory
downstream latency

Mitigation: per-tenant rate limiting, concurrency caps, queues, and backpressure.

5) “Helpful logs” → data leak

Tool arguments and tool responses often contain secrets, tokens, or private data. If you log everything, you’ve built an involuntary data lake.

Mitigation: structured + redacted logging, security logging guidelines, and minimal retention. [11][12]

Hardening layer 1: identity and authorization

If you run Streamable HTTP, assume:

multiple clients
untrusted networks
tokens will leak eventually

MCP’s architecture guidance recommends standard HTTP authentication methods and mentions OAuth as a recommended way to obtain tokens for remote servers. [2][3]

Practical rules

Authenticate every request.
Use bearer tokens or mTLS depending on environment.
Authorize per tool.
“Authenticated” ≠ “allowed to run delete_everything”.
Prefer short-lived tokens and rotate them. [12]
Multi-tenant? Put the tenant identity into:
- auth token claims, or
- an explicit, validated tenant header (signed), then
- enforce it everywhere.

Go pattern: a minimal auth middleware skeleton (HTTP transport)

This is not a full MCP implementation, just the hardening pattern you’ll wrap around your MCP handler.

// Pseudocode-ish middleware skeleton. Replace verifyToken with your auth logic.
func authMiddleware(next http.Handler) http.Handler {
 return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 token := strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ")
 if token == "" {
 http.Error(w, "missing auth", http.StatusUnauthorized)
 return
 }

 ident, err := verifyToken(r.Context(), token) // includes tenant + scopes
 if err != nil {
 http.Error(w, "invalid auth", http.StatusUnauthorized)
 return
 }

 ctx := context.WithValue(r.Context(), ctxKeyIdentity{}, ident)
 next.ServeHTTP(w, r.WithContext(ctx))
 })
}

Key point: authorization should happen after you parse the requested tool name, but before you execute anything.

Hardening layer 2: tool contracts that resist ambiguity

Most MCP tool failures are self-inflicted: tool interfaces are too vague.

Design tools like production APIs

Bad tool signature:

run(command: string)

Better:

run_command(program: enum, args: string[], cwd: string, timeout_ms: int, dry_run: bool)

Why it’s better:

forces structure
allows you to enforce allowlists
gives you timeouts and safe defaults

Add a “preview → apply” flow for risky tools

For any tool that writes data or triggers side effects, do a two-step approach:

plan_* returns a machine-readable plan + a plan_id
apply_* requires plan_id and optional user confirmation token

This mirrors how we run infra changes (plan/apply) and dramatically reduces accidental blast radius.

Hardening layer 3: budgets and backpressure

Production systems are budget systems.

If you don’t set explicit budgets, your MCP server will eventually allocate them for you via outages.

Budget checklist

Server timeouts (header read, request read, write, idle)
Request body caps
Outbound timeouts to dependencies
Concurrency caps per tool and per tenant
Rate limits per tenant and per identity
Queue limits (bounded channels) to avoid memory blowups
Circuit breaking for flaky downstream dependencies

Go: server timeouts are not optional

Go’s net/http provides explicit server timeouts; leaving them at zero is a common footgun. [6][7]

srv := &http.Server{
 Addr: ":8080",
 Handler: handler, // your MCP handler + middleware
 ReadHeaderTimeout: 5 * time.Second,
 ReadTimeout: 30 * time.Second,
 WriteTimeout: 30 * time.Second,
 IdleTimeout: 60 * time.Second,
}
log.Fatal(srv.ListenAndServe())

Go: propagate cancellation everywhere with `context`

context.Context is the backbone of “structured concurrency” in Go: deadlines and cancellation signals flow through your call stack. [8][9]

Rule: every tool execution must accept a context.Context, and every outbound call must honor it.

func (s *Server) toolCall(ctx context.Context, req ToolRequest) (ToolResponse, error) {
 ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
 defer cancel()

 // ... outbound calls use ctx
 return s.integration.Do(ctx, req)
}

Go: per-tenant rate limiting with `x/time/rate`

golang.org/x/time/rate implements a token bucket limiter. [9]

type limiters struct {
 mu sync.Mutex
 m map[string]*rate.Limiter
}

func (l *limiters) get(key string) *rate.Limiter {
 l.mu.Lock()
 defer l.mu.Unlock()
 if l.m == nil { l.m = map[string]*rate.Limiter{} }
 if lim, ok := l.m[key]; ok { return lim }

 // Example: 5 req/sec with bursts up to 10
 lim := rate.NewLimiter(5, 10)
 l.m[key] = lim
 return lim
}

func rateLimitMiddleware(lims *limiters, next http.Handler) http.Handler {
 return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 ident := mustIdentity(r.Context())
 if !lims.get(ident.TenantID).Allow() {
 http.Error(w, "rate limited", http.StatusTooManyRequests)
 return
 }
 next.ServeHTTP(w, r)
 })
}

Backpressure: choose a policy

When you’re overloaded, you need a policy. Pick one explicitly:

Fail fast with 429 / “busy” (simplest, safest)
Queue with bounded depth (more complex; must cap memory)
Degrade by disabling expensive tools first

The “fail fast” approach is often correct for tool gateways.

Hardening layer 4: safe networking and SSRF containment

If any tool can fetch a user-provided URL or call a user-influenced endpoint, SSRF is on the table. [10]

SSRF containment strategies that actually work

OWASP’s SSRF guidance boils down to a few themes: don’t trust user-controlled URLs, use allowlists, and enforce network controls. [10]

In practice, for MCP servers:

Prefer allowlists over blocklists.
“Only these domains” beats “block internal IPs.” Attackers are creative.
Resolve and validate IPs before dialing.
DNS can be weaponized. Validate the final destination IP (and re-validate on redirects).
Disable redirects or re-validate each hop.
Redirect chains are SSRF’s favorite tool.
Enforce egress policy at the network layer too.
Kubernetes NetworkPolicies / firewall rules are your last line of defense.

Go pattern: an outbound HTTP client with strict timeouts

client := &http.Client{
 Timeout: 10 * time.Second, // whole request budget
 Transport: &http.Transport{
 Proxy: http.ProxyFromEnvironment,
 DialContext: (&net.Dialer{
 Timeout: 5 * time.Second,
 KeepAlive: 30 * time.Second,
 }).DialContext,
 TLSHandshakeTimeout: 5 * time.Second,
 ResponseHeaderTimeout: 5 * time.Second,
 ExpectContinueTimeout: 1 * time.Second,
 MaxIdleConns: 100,
 IdleConnTimeout: 90 * time.Second,
 },
}

Then wrap URL validation around any request creation. Keep it boring and strict.

Hardening layer 5: observability without leaking secrets

Telemetry is how you prove:

you’re within budgets
tools behave as expected
failures are localized
incidents can be diagnosed without “ssh and guess”

But logging is also where teams accidentally leak sensitive data.

OWASP’s logging guidance emphasizes logging that supports detection/response while avoiding sensitive data exposure. [11] Pair that with secrets management discipline. [12]

What to measure (minimum viable MCP telemetry)

Counters

tool_calls_total{tool, tenant, status}
auth_failures_total{reason}
rate_limited_total{tenant}

Histograms

tool_latency_seconds{tool}
outbound_latency_seconds{dependency}

Gauges

in_flight_tool_calls{tool}
queue_depth{tool}

Trace boundaries

Instrument:

request → tool routing
tool execution span
downstream calls span

OpenTelemetry’s Go docs show how to add instrumentation and emit traces/metrics. [13]

Logging rules that save you later

Use structured logging (JSON).
Add correlation IDs (trace IDs) to logs.
Redact:
- Authorization headers
- tokens
- cookies
- tool payload fields known to contain secrets
Log events, not raw payloads:
- “tool X called”
- “resource Y read”
- “write operation requested (dry_run=true)”

Audit logs

For high-impact tools, write an append-only audit record:
- who (identity)
- what (tool + parameters summary)
- when
- result (success/failure)
- plan_id / idempotency_key

Audit logs should be treated as security data.

Hardening layer 6: versioning and rollout discipline

MCP uses string-based version identifiers like YYYY-MM-DD to represent the last date of backwards-incompatible changes. [4]

That’s helpful, but it doesn’t solve the operational problem:

clients upgrade at different times
schema changes drift
hosts differ in which capabilities they support

Practical compatibility rules

Pin your server’s supported protocol version and expose it in health or diagnostics.
Add contract tests that run against:
- one “current” client
- one “previous” client version
Support additive changes first:
- new tools
- new optional fields
Use feature flags for risky tools.

Rollout like a platform team

Canaries for remote servers
“Shadow mode” for new tools (log what would happen)
Slow ramp with budget monitoring

A production checklist

If you’re building (or inheriting) an MCP server, run this checklist:

Safety

Tool contracts are structured (no free-form “do anything” strings).
Every tool has a safe default (dry_run=true, limit required, etc.).
Destructive tools require a plan/apply step (or explicit confirmation gates).
Tool inputs are validated and bounded (length, ranges, enums).

Identity & access

Remote transport requires authentication and per-tool authorization.
Tokens are short-lived and rotated; secrets are not in source control. [12]
Tenant identity is enforced at every access point (not “best effort”).

Budgets & resilience

HTTP server timeouts are configured. [6][7]
Outbound clients have timeouts and connection limits.
Rate limiting exists per tenant/identity. [9]
Concurrency caps exist per tool; overload behavior is explicit (fail fast / queue).
Retries are bounded and idempotent where side effects exist.

Networking

URL fetch tools have allowlists and SSRF protections. [10]
Redirect policies are explicit (disabled or re-validated).
Egress is constrained at the network layer (not only in code).

Observability

Metrics cover tool calls, latency, errors, and rate limiting.
Tracing exists across tool execution and downstream calls. [13]
Logs are structured, correlated, and redacted. [11]
Audit logging exists for high-impact tools.

Operations

Health checks and readiness checks exist.
Configuration is explicit and validated on startup.
Versioning strategy is documented and tested. [4]

References

Model Context Protocol (MCP) Specification (version 2025-11-25): https://modelcontextprotocol.io/specification/2025-11-25
MCP Architecture Overview (participants, transports, concepts): https://modelcontextprotocol.io/docs/learn/architecture
MCP Transport details (Streamable HTTP transport overview): https://modelcontextprotocol.io/specification/2025-03-26/basic/transports
MCP Versioning: https://modelcontextprotocol.io/specification/versioning
JSON-RPC 2.0 Specification: https://www.jsonrpc.org/specification
Go net/http package documentation: https://pkg.go.dev/net/http
Cloudflare: “The complete guide to Go net/http timeouts”: https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/
Go context package documentation: https://pkg.go.dev/context
Go x/time/rate documentation: https://pkg.go.dev/golang.org/x/time/rate
OWASP SSRF Prevention Cheat Sheet / SSRF category references:

OWASP Logging Cheat Sheet (security-focused logging guidance): https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
Secrets management guidance:

OWASP Secrets Management Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
Kubernetes “Good practices for Kubernetes Secrets”: https://kubernetes.io/docs/concepts/security/secrets-good-practices/

OpenTelemetry Go instrumentation docs: https://opentelemetry.io/docs/languages/go/instrumentation/

Chapter 9: Security & Sensitive Data: Sanitize, Don't Paste Secrets

Thu, 29 Jan 2026 21:00:00 -0500

Series: LLM Development Guide

Chapter 9 of 16

Previous: Chapter 8: Choosing the Right Model: Capability Tiers, Not Hype

Next: Chapter 10: Stop Rules + Pitfalls: When to Upgrade, Bail, or Go Manual

What you’ll be able to do

You’ll be able to use LLMs without doing something reckless:

Apply a concrete “never paste” list.
Sanitize code, config, and logs into safe examples.
Add a verification step so you don’t ship secrets.

TL;DR

Assume anything you paste could be logged or retained.
If you wouldn’t publish it publicly, don’t paste it.
Replace real values with placeholders.
Sanitize logs aggressively.
Verify your workspace for leaked secrets before you commit.

The core principle
Never paste list
Sanitization patterns
Verification
Failure modes

The core principle

Assume anything you send to an LLM could be stored.

Even with enterprise offerings, policies change. Check vendor policy as of 2026-02-14 (and your organization’s approved tools list) before using any tool with internal data.

Never paste list

Do not paste:

Credentials: API keys, tokens, passwords, private keys.
PII: customer names, emails, addresses, health data.
Production data: real records, full dumps, support tickets.
Security configs: firewall rules, IAM policies, internal IPs.
Proprietary secrets: unreleased product details, trade secrets.

Use the “Would I post this publicly?” test.

Sanitization patterns

Replace sensitive values with descriptive placeholders.

Go example:

// Before (do not paste)
// db, err := sql.Open("postgres", "host=prod-db.internal user=admin password=SuperSecret123 dbname=customers")

// After (safe to paste)
db, err := sql.Open("postgres", "host=DATABASE_HOST user=DATABASE_USER password=DATABASE_PASSWORD dbname=DATABASE_NAME")
if err != nil {
 return err
}

YAML example:

# Before (do not paste)
# data:
# api-key: YWN0dWFsLWFwaS1rZXktaGVyZQ==

# After (safe to paste)
data:
 api-key: <BASE64_ENCODED_API_KEY>
 webhook-secret: <BASE64_ENCODED_WEBHOOK_SECRET>

Logs:

Remove emails.
Replace internal hostnames.
Replace IPs with documentation ranges (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24).

Verification

Before you paste or commit, search your workspace for obvious secret patterns.

These commands are noisy, but useful:

# High-signal patterns.
rg -n "(AKIA[0-9A-Z]{16}|BEGIN (RSA|OPENSSH) PRIVATE KEY|xox[baprs]-|ghp_[A-Za-z0-9]{36})" . || true

# Common key/value names.
rg -n "(?i)(api[_-]?key|secret|token|password)\s*[:=]" . || true

# Emails (often indicates logs or real data got copied).
rg -n "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}" . || true

# Check staged changes specifically.
git diff --cached

Expected results:

No obvious credentials or private keys appear in diffs.
If matches exist, sanitize and regenerate the example.

Failure modes

Sharing real logs that contain tokens in URLs.
Copying a Kubernetes Secret verbatim.
Letting an IDE plugin send your whole file without noticing.
Assuming “enterprise” means “no risk” without verifying current policy.

Continue -> Chapter 10: Stop Rules + Pitfalls: When to Upgrade, Bail, or Go Manual

Chapter 8: Choosing the Right Model: Capability Tiers, Not Hype

Tue, 27 Jan 2026 19:00:00 -0500

Series: LLM Development Guide

Chapter 8 of 16

Previous: Chapter 7: Large Projects with Phase Documents + Implementation Prompts

Next: Chapter 9: Security & Sensitive Data: Sanitize, Don’t Paste Secrets

What you’ll be able to do

You’ll be able to pick a model and interface deliberately:

Use capability tiers instead of memorizing brand names.
Upgrade quickly when quality is the bottleneck.
Avoid wasting flagship models on structured boilerplate.

TL;DR

Treat model choice as a cost-of-mistakes problem.
Use flagship models for planning, debugging, and high-stakes decisions.
Use mid-tier models for implementation with strong references.
Use fast/cheap models for boilerplate and simple transformations.
If you’ve spent ~10 minutes fighting output quality, upgrade or shrink scope.

As-of note

As of 2026-02-14, model names, pricing, and product policies change frequently. Prefer tier-based guidance, and verify vendor policies directly before using tools with sensitive data.

The capability tiers
Task-to-tier mapping
Red flags: upgrade now
A selection checklist
Verification

The capability tiers

Think in tiers:

Flagship: best reasoning and instruction-following for novel work.
Mid-tier: strong general performance for structured work with references.
Fast/cheap: good for simple tasks, higher error rate on complex reasoning.

This framing stays useful even when names change.

Task-to-tier mapping

Use flagship for:

Planning and architecture.
Debugging complex failures.
Security-sensitive review.
Anything where mistakes are expensive.

Use mid-tier for:

Implementation that follows existing patterns.
Refactors with clear examples.
Writing tests when the behavior is already defined.

Use fast/cheap for:

Syntax lookups.
Boilerplate you will review.
Mechanical transformations.

Red flags: upgrade now

Upgrade when you see:

The model repeats the same misunderstanding.
Output ignores constraints.
“Looks right” code fails in tests.
You are on the third prompt iteration for the same unit.

The cheapest model is the one that gets you to a correct verified change with the least total time.

A selection checklist

Before you start, answer:

Is this novel or pattern-following?
Do I have reference implementations?
What is the cost of mistakes?
Is this structured or ambiguous?
Am I debugging or implementing?

If uncertain:

Start with flagship for planning.
Drop to mid-tier once you have a stable pattern and good references.

Verification

A practical way to keep this from being hand-wavy is to force a written decision per phase.

Create a small note file per task:

mkdir -p work-notes

cat > work-notes/model-selection.md <<'MD'
# Model Selection (Per Task)

## Task
<What are we doing?>

## Risk
- Cost of mistakes:
- Can I review the output competently?

## References
- <Paths to reference implementations>

## Model decision
- Tier: <flagship|mid-tier|fast>
- Why:
- When to upgrade:

## Outcome
- Did we upgrade?
- What broke / what worked:
MD

Expected result:

You can justify the model choice in one minute.
You have a trigger for upgrading when output quality is the bottleneck.

Continue -> Chapter 9: Security & Sensitive Data: Sanitize, Don’t Paste Secrets

Chapter 7: Large Projects with Phase Documents + Implementation Prompts

Mon, 26 Jan 2026 20:00:00 -0500

Series: LLM Development Guide

Chapter 7 of 16

Previous: Chapter 6: Scaling the Workflow: Phases, Parallelism, Hygiene

Next: Chapter 8: Choosing the Right Model: Capability Tiers, Not Hype

What you’ll be able to do

You’ll be able to run large, multi-phase delivery with less drift by introducing two explicit artifacts:

A phase specification document that defines scope, dependencies, files, and exit criteria.
A phase implementation prompt document that defines the prompt-by-prompt execution contract.
A repeatable operating cadence for execution, verification, and commits.

TL;DR

Large projects fail when a single prompt tries to carry the whole implementation plan.
Use one phase spec and one implementation-prompt file per sub-phase.
Execute prompts sequentially; do not continue if build/vet/test gates fail.
Keep context loading explicit for each prompt.
For copy/paste templates, use Chapter 13: Templates + Checklists: The Copy/Paste Kit .

Why this pattern exists
The two-document system
Worked example: a multi-phase engineering initiative
Execution protocol for prompt files
Verification
Failure modes

Why this pattern exists

For a one-day task, a plan plus one execution prompt is usually enough.

For multi-week work, that breaks down:

Context gets too large and detail gets dropped.
Sessions diverge when constraints are implied instead of written.
Verification becomes optional instead of required.
Commits become large and hard to review.

The fix is to treat phase docs and implementation prompt docs as first-class project artifacts.

The two-document system

For each sub-phase, create two files.

1) Phase spec document

Purpose: define what this sub-phase must accomplish and how completion is validated.

Typical sections:

Status, dependency, and migration notes.
Design rationale (why this slice exists now).
Tasks grouped by prompt number.
Files: new, modified, and referenced-only.
Exit criteria with concrete commands and expected results.
Progress notes placeholder.

2) Phase implementation prompt document

Purpose: define exactly how execution happens, prompt by prompt.

Each prompt should include:

Context files to load (small, explicit list).
Task details: signatures, interfaces, constraints.
Quality gates and required verification commands.
Stop condition: do not proceed until the current prompt passes.

A useful pattern is to couple one prompt to one logical implementation unit.

Worked example: a multi-phase engineering initiative

Assume you are delivering a new runtime capability over six weeks.

You split work into:

Phase A: contracts and types.
Phase B: core implementation.
Phase C: API and integration points.
Phase D: tests and validation.
Phase E: observability and rollout safety.

For Phase B, your phase spec might look like this:

# Phase B - Core Implementation

## Status
Planned

## Depends on
Phase A

## Design rationale
Phase B isolates core behavior behind the contracts from Phase A.
This prevents API and infrastructure concerns from polluting the core logic.

## Tasks
### Prompt 1
- Implement core orchestration types and constructor.

### Prompt 2
- Implement main execution method with deterministic error paths.

### Prompt 3
- Add unit tests for success and failure branches.

## Files
### New
- internal/core/runtime.go
- internal/core/runtime_test.go

### Modified
- internal/core/types.go

### Referenced (read-only)
- internal/contracts/interfaces.go

## Exit criteria
- [ ] `go build ./internal/core/...` exits 0
- [ ] `go vet ./internal/core/...` exits 0
- [ ] `go test ./internal/core/...` exits 0
- [ ] No unchecked returned errors

## Progress notes

Now pair it with a Phase B implementation prompt file:

# Phase B - Implementation Prompts

## Prompt 1 of 3: Runtime skeleton

Context files to load:
- docs/phases/PHASEB.md
- internal/contracts/interfaces.go
- internal/core/types.go
- README.md

Task:
- Create `internal/core/runtime.go` with constructor and public methods.

Constraints:
- Do not change files outside listed scope.
- Handle all returned errors explicitly.
- Keep methods short enough to remain reviewable.

Verification:
- `go build ./internal/core/...`
- `go vet ./internal/core/...`

Stop rule:
- Do not proceed to Prompt 2 until both commands pass.

This is intentionally boring. Boring is what scales.

Execution protocol for prompt files

Use the same cadence for every prompt in a sub-phase:

Load only listed context files.
Execute exactly one prompt.
Update work notes (decisions, assumptions, blockers, next step).
Run required verification gates.
Commit one logical unit.
Move to the next prompt.

Suggested commit discipline:

One commit per prompt when prompts are independent.
One commit per tightly coupled prompt pair when separation creates broken intermediate states.
Message format should state scope and intent clearly.

When prompt counts are high, add a completion table in work notes:

## Prompt progress
- [x] Prompt 1
- [x] Prompt 2
- [ ] Prompt 3
- [ ] Prompt 4

Verification

You can verify this system is functioning with mechanical checks.

# All phase specs have exit criteria.
rg -n "^## Exit criteria" docs/phases/PHASE*.md

# All prompt docs define context loading and verification.
rg -n "^Context files to load:|^Verification:" docs/phases/*-PROMPT.md

# Work notes track progression.
rg -n "^## Prompt progress|^## Session log" work-notes || true

Expected results:

Every phase spec has explicit exit criteria.
Every prompt file defines context and verification.
Session state is recoverable without re-explaining the whole project.

Failure modes

Phase docs describe architecture but skip executable gates.
Prompt docs are too broad (“implement phase”) and lose determinism.
Prompts proceed despite failing verification.
Context file lists are bloated and include unrelated material.

If this starts happening, shrink prompt scope and tighten exit criteria before continuing.

Continue -> Chapter 8: Choosing the Right Model: Capability Tiers, Not Hype

Chapter 6: Scaling the Workflow: Phases, Parallelism, Hygiene

Sun, 25 Jan 2026 18:00:00 -0500

Series: LLM Development Guide

Chapter 6 of 16

Previous: Chapter 5: The Execution Loop: Review Discipline + Commit Discipline

Next: Chapter 7: Large Projects with Phase Documents + Implementation Prompts

What you’ll be able to do

You’ll be able to take the same workflow and scale it up without chaos:

Split work into phases that do not overlap on files.
Run parallel sessions or agents safely.
Decide when artifacts stay local vs. get committed.

TL;DR

Scale by splitting phases, not by writing bigger prompts.
Keep phase files small (rule of thumb: under ~200 lines).
Parallel work requires clean boundaries and explicit interfaces.
Decide up front whether plan/, prompts/, work-notes/ live in git.

When to sub-phase
Parallel execution requirements
Repository hygiene
Verification
Gotchas

When to sub-phase

Sub-phase when:

A phase touches too many files.
The phase cannot be verified independently.
The phase depends on decisions that are not written down.

Example layout:

plan/
 phase-1a-analysis.md
 phase-1b-design.md
 phase-2a-scaffold.md
 phase-2b-core-impl.md
 phase-3-validation.md

Keep each phase “one session friendly”: small enough to complete (or at least checkpoint) in 1 to 2 sessions.

At this point, many teams benefit from formalizing each sub-phase with two files: a phase spec and a phase implementation prompt file. The next chapter walks through that pattern in detail.

Parallel execution requirements

Parallel work is possible, but only if you make boundaries explicit.

Requirements:

No overlapping files between parallel phases.
Explicit interfaces (types, APIs, data shapes) written down.
A merge plan (who rebases, who resolves conflicts, how often you sync).

A simple pattern:

Phase A defines interfaces and contracts.
Phase B implements with those interfaces.
Phase C adds tests and validation.

Repository hygiene

Decide whether artifacts are local scaffolding or part of the repo.

Common default:

Keep plan/, prompts/, work-notes/ local (gitignored).

Commit them deliberately when they become:

Long-lived docs.
Reusable templates.
Onboarding material.

If you commit them, consider moving to docs/ and editing for humans.

Verification

These checks help you catch “phases are too big” early:

# Find phase files that are getting too large.
# (This uses line count as a blunt proxy.)
find plan -type f -name '*.md' -maxdepth 2 -print0 | xargs -0 wc -l | sort -n

# Find prompts that do not reference work notes.
rg -n "work-notes/" prompts || true

# Find phases that might overlap on files (manual review).
# Start by listing deliverables per phase in each prompt doc.
rg -n "^## Deliverables" -n prompts

Gotchas

Parallelization without clean boundaries just creates merge conflicts faster.
If you don’t define interfaces early, later phases stall.
If artifacts are committed, treat them like code: review, version, maintain.

Continue -> Chapter 7: Large Projects with Phase Documents + Implementation Prompts

When Management Layers Become Latency

Sat, 24 Jan 2026 10:30:00 -0500

Note on examples: The scenarios below are anonymized composites. This isn’t “management bad.” Good management is an accelerator. The problem is when management becomes layers of translation between reality and decisions.

Why this matters

In production systems, adding hops between a request and a response increases latency, failure modes, and debugging time.

Organizations behave the same way.

When engineering work flows through too many intermediary layers - tech leads, scrum masters, managers, senior managers, project managers, directors, senior directors, VPs, and beyond - the organization starts to exhibit the same symptoms as an over-proxied network:

long lead times
lost context (“telephone game” requirements)
local optimization (everyone looks busy; value doesn’t move)
coordination overhead that scales faster than delivery
engineers feeling like nothing they build reaches production

The painful part is that the org can look healthy on paper (status is green, roadmaps are full) while the product fails to meet real expectations.

This article is about the mechanics behind that failure - and the replacement patterns that restore flow.

TL;DR

Layers create handoffs. Handoffs create queues. Queues create lead time.
More roles don’t automatically increase throughput; coordination cost can dominate (Brooks’s Law). [6]
Fast flow requires end-to-end ownership with minimal handoffs (stream-aligned teams). [3][4]
Measure outcomes at the system level (DORA metrics), not “activity” (story points, number of meetings). [1]
Don’t turn metrics into targets (Goodhart’s Law). [7]
Burnout often rises when delivery is painful and risky; improving delivery capability predicts lower burnout. [2][8]

Pattern 1: Translation layers replace direct truth
Pattern 2: Status becomes the work
Pattern 3: “More people” is treated like a throughput solution
Pattern 4: Projectization and temporary teams
Pattern 5: Governance by meeting instead of guardrail
Pattern 6: Metrics as targets
Pattern 7: Engineers are abstracted away from production
Replacement patterns that work
Verification: how you know the org is healing
A practical checklist
References

Pattern 1: Translation layers replace direct truth

What it looks like

A customer need or operational pain moves through a chain:

customer -> product -> program -> project -> delivery manager -> engineering manager -> tech lead -> engineers

By the time it arrives at the team, it’s been translated multiple times and often loses:

the actual user story
the constraints
the real priority
the “why”

Why it exists

Layering feels safe:

fewer people “bother” engineers
leaders get curated information
decision makers see clean narratives

The hidden tax

Misalignment becomes normal.
Engineers build the wrong thing efficiently.
Product expectations aren’t met, not because engineers can’t build - but because the input signal is degraded.

The replacement pattern

Shorten the feedback loop.

Ensure teams have direct access to:
customer signals (support tickets, usage, interviews)
operational signals (incidents, latency, error budgets)
Make the “why” non-optional: put it in the ticket, the PRD, and the kickoff.

If a team can’t explain “why this exists,” it shouldn’t ship yet.

Pattern 2: Status becomes the work

What it looks like

Organizations that struggle to ship often compensate with:

more meetings
more dashboards
more decks
more “alignment sessions”

The output looks like progress, but the production system doesn’t change.

Why it exists

When uncertainty is high, visibility is comforting.

The hidden tax

Attention becomes scarce.
Engineers fragment into “meeting responders.”
Work becomes multi-tasked across too many initiatives (WIP explosion).

The replacement pattern

Reduce status overhead by making the system visible:

CI/CD dashboards
production telemetry
an engineering scorecard based on system outcomes (not activity)

DORA’s metrics are widely used as system-level indicators for delivery performance: deployment frequency, lead time, change failure rate, and time to restore service. [1]

Pattern 3: “More people” is treated like a throughput solution

What it looks like

A late initiative triggers:

new managers
new project managers
new engineers
more coordination rituals

Why it exists

It’s intuitive: more people should mean more output.

The hidden tax

Software delivery has a coordination component. Adding people increases communication paths, onboarding, and synchronization.

Brooks’s Law captures this succinctly: adding manpower to a late software project can make it later. [6]

The replacement pattern

Before adding headcount, reduce coordination load:

clarify ownership
shrink scope to a thin vertical slice
eliminate handoffs
stabilize requirements long enough to ship

Then scale with:

duplication (more teams owning similar streams)
platform leverage (paved roads), not more meetings

Pattern 4: Projectization and temporary teams

What it looks like

Engineers are repeatedly reorganized into short-lived “project teams,” and after delivery they are moved again.

Why it exists

Projects are easy to budget, track, and narrate.

The hidden tax

Temporary teams produce:

fragile ownership
weak operability
“throw it over the wall” incentives

Fast flow requires teams that own outcomes end-to-end with minimal handoffs.

Team Topologies describes stream-aligned teams as owning a slice of value end-to-end with no handoffs. [3][4]

The replacement pattern

Prefer stable teams aligned to a value stream (product/service), with:

clear ownership
operational responsibility (“you build it, you run it”)
direct feedback from users and production

Pattern 5: Governance by meeting instead of guardrail

What it looks like

Instead of “how do we make safe delivery easy,” governance becomes:

approval steps
committees
sign-off chains

Why it exists

Risk is real, and leaders want control.

The hidden tax

Humans are expensive control planes:

slow
inconsistent
difficult to audit at scale

The replacement pattern

Convert rules into guardrails:

policy-as-code
templates
paved paths
automated checks in CI/CD

This is how you scale safety without scaling meetings.

Pattern 6: Metrics as targets

What it looks like

Teams are pressured to hit:

story points
“velocity”
number of deployments
“percent complete”
tickets closed

Then behavior adapts to the metric.

Why it exists

Leaders need a dashboard.

The hidden tax

When a measure becomes a target, it can stop being a good measure (Goodhart’s Law). [7]

Examples:

inflate points
ship low-value changes to increase deploy count
avoid hard work because it hurts “throughput”

The replacement pattern

Use metrics diagnostically at the system level (not as individual KPIs).

If you adopt DORA metrics, use them to identify constraints and improve flow - not as quarterly targets for teams. [1][9]

Pattern 7: Engineers are abstracted away from production

What it looks like

A team builds a system, but:

another team deploys it
another team runs it
another team handles incidents
another team owns the roadmap

Engineers eventually conclude: “Nothing I build actually ships.”

Why it exists

Specialization can be useful, but excessive separation breaks feedback loops.

The hidden tax

teams don’t learn from production
quality declines because consequences are indirect
“deployment pain” rises: shipping becomes stressful and disruptive

DORA describes deployment pain as fear/anxiety around deploying and links it to poorer delivery performance and culture. [8] DORA also notes continuous delivery predicts lower levels of burnout and reduces deployment pain. [2]

The replacement pattern

Re-connect engineers to production:

give teams operational ownership for what they build
make telemetry and incident review part of engineering
reduce fear by making releases small, frequent, and observable

Replacement patterns that work

These are the patterns I’ve seen consistently restore delivery flow without chaos.

1) Clarify decision rights (and keep them close to the work)

One accountable owner per initiative (not “everyone is accountable”)
Engineers participate in tradeoff decisions early (scope, sequencing, risk)

2) Design teams for flow (not for org charts)

Organizations build systems that mirror their communication structures (Conway’s Law). [5] If your org is siloed and layered, your architecture often becomes siloed and layered too.

Design teams so the desired architecture is the path of least resistance.

3) Prefer stream-aligned teams + platform leverage

Stream-aligned teams own outcomes end-to-end (no handoffs). [3][4]
Platform teams reduce cognitive load by providing paved roads (auth, telemetry, CI/CD). [4]

4) Replace “alignment meetings” with shared artifacts

one-page decision records
clear “definition of done”
demos that show working software in a real environment

5) Turn delivery into a calm, repeatable process

When delivery is painful, people add layers to manage fear. Fix the source:

tests
automation
progressive delivery
observable releases

That’s how you reduce burnout sustainably. [2][8]

Verification: how you know the org is healing

Don’t rely on vibes. Use evidence.

Delivery outcomes (system-level)

Start with DORA metrics to track flow and stability. [1]

Product outcomes

adoption (are users actually using the thing?)
retention (does usage persist?)
reduced operational toil (do incidents go down?)

Team outcomes

fewer emergency escalations
fewer “status-only” meetings
improved on-call experience (lower deployment pain) [8]

If lead time drops but burnout rises, you probably “optimized the dashboard” instead of the system (see Goodhart). [7]

A practical checklist

If your org feels “management-heavy,” try this in order:

Reduce translation layers

Put engineers in the room (or thread) with real users/operators at least weekly.
Require the “why” to be written and reviewed before build starts.

Reduce handoffs

Map the value stream and count handoffs.
Remove one handoff per quarter; make it a goal.

Reduce WIP

Limit concurrent initiatives per team.
Finish before starting.

Convert meetings into guardrails

Replace approvals with automated checks where possible.
Create paved paths so the safe way is the easy way.

Reconnect teams to production

Teams own what they ship.
Tie incident learning back to design decisions.
Make releases smaller and more frequent.

References

[1] DORA - “DORA’s software delivery performance metrics (guide)”. https://dora.dev/guides/dora-metrics/ [2] DORA - “Capabilities: Continuous delivery” (notes relationship to burnout and deployment pain). https://dora.dev/capabilities/continuous-delivery/ [3] Team Topologies - “Key Concepts” (stream-aligned teams; no handoffs). https://teamtopologies.com/key-concepts [4] IT Revolution - “The Four Team Types from Team Topologies” (stream-aligned teams own end-to-end). https://itrevolution.com/articles/four-team-types/ [5] Splunk - “Conway’s Law Explained” (systems mirror communication structures; includes original quote). https://www.splunk.com/en_us/blog/learn/conways-law.html [6] Brooks’s Law (coined in The Mythical Man-Month): “Adding manpower to a late software project makes it later.” https://en.wikipedia.org/wiki/Brooks%27s_law [7] CNA - “Goodhart’s Law” (when a measure becomes a target, it ceases to be a good measure). https://www.cna.org/analyses/2022/09/goodharts-law [8] DORA - “Capabilities: Well-being” (deployment pain and its relationship to performance/culture). https://dora.dev/capabilities/well-being/ [9] SEI (CMU) - “How to Misuse and Abuse DORA Metrics” (metric anti-patterns). https://www.sei.cmu.edu/library/how-to-misuse-and-abuse-dora-metrics/

Chapter 5: The Execution Loop: Review Discipline + Commit Discipline

Fri, 23 Jan 2026 16:00:00 -0500

Series: LLM Development Guide

Chapter 5 of 16

Previous: Chapter 4: Work Notes: External Memory + Running Log

Next: Chapter 6: Scaling the Workflow: Phases, Parallelism, Hygiene

What you’ll be able to do

You’ll be able to run an LLM through implementation work in a way that stays reviewable:

One logical unit at a time.
Verification before claiming “done”.
Atomic commits with clear intent.
Notes updated as part of the loop.

TL;DR

Never skip: update notes, verify, propose commit, review.
A “logical unit” is the smallest change that is independently reviewable.
Treat LLM output like junior output: it needs review.
Keep commits small to make rollback cheap.

The execution loop
What counts as a logical unit
Commit discipline
Verification
Failure modes

The execution loop

This loop is intentionally repetitive:

Load prompt + work-notes
-> implement one logical unit
-> update work-notes
-> verify
-> propose commit
-> you review
-> commit
-> repeat

If you skip the middle steps, your “agent” becomes a vibes-based code generator.

What counts as a logical unit

A logical unit is a change that:

Has a clear purpose.
Can be verified.
Can be reviewed in isolation.
Does not leave the repo in a half-broken state.

Examples:

Add one Kubernetes template file.
Add one Go type + its tests.
Add one API endpoint handler.

Non-examples:

“Implement everything for phase 2”.
“Half a refactor”.
“Quick fixes”.

Commit discipline

Use a consistent commit format so you can scan history later.

Example commit message:

feat(helm): add service template for metrics-gateway

- Exposes port 9090 as ClusterIP
- Follows event-processor naming and labels
- No ingress in this commit

Refs: work-notes/phase-2b-core-resources.md

In prompts, require the model to:

Summarize what changed.
Explain why.
Propose a message.
Wait for approval.

Verification

Verification is context-specific, but the shape is consistent:

Build.
Test.
Lint.
Render config.

Generic verification commands you can adapt:

# Make sure you know what is staged and what is not.
git status --porcelain

git diff

# Run the repo's verification gates.
# Replace these with your actual commands.
go test ./...

# After the commit, ensure you're clean.
git status --porcelain

Expected results:

Before committing, git diff matches the logical unit scope.
After committing, git status --porcelain is empty.
Tests and other gates exit 0.

Failure modes

The model keeps “just one more change”-ing.
- Fix: put explicit stop points in the prompt.
You don’t verify.
- Fix: add verification commands to plan and prompt docs.
Commits include unrelated changes.
- Fix: shrink the logical unit and split the work.
You approve output you can’t review.
- Fix: stop and do it manually or bring in a reviewer.

Continue -> Chapter 6: Scaling the Workflow: Phases, Parallelism, Hygiene

Chapter 4: Work Notes: External Memory + Running Log

Wed, 21 Jan 2026 14:00:00 -0500

Series: LLM Development Guide

Chapter 4 of 16

Previous: Chapter 3: Prompt Documents: Prompts That Survive Sessions

Next: Chapter 5: The Execution Loop: Review Discipline + Commit Discipline

What you’ll be able to do

You’ll be able to keep multi-session work consistent by maintaining work notes that:

Preserve the model’s working state outside the chat.
Capture decisions and rationale for later review.
Make handoffs possible.
Provide a deterministic “resume” prompt.

TL;DR

LLMs have no durable memory. If it’s not written down, it doesn’t exist next session.
Mirror work-notes/ files to your phases exactly.
Track: status, decisions, assumptions, open questions, session log, commits.
In your prompts, require the model to update notes before moving forward.

Directory alignment
A work-notes template you can paste
Session start and session end prompts
Verification
Gotchas

Directory alignment

Keep your three directories aligned so you can load one phase without dragging unrelated context into the session:

plan/phase-2a-scaffolding.md
prompts/phase-2a-scaffolding.md
work-notes/phase-2a-scaffolding.md

This makes sessions resumable and makes parallel work possible.

A work-notes template you can paste

# Phase <X> - <Phase Name>

## Status
- [ ] Not started
- [ ] In progress
- [ ] Blocked
- [ ] Complete

## Decisions
- <Decision>: <Rationale>

## Assumptions
- <Assumption>

## Open questions
- [ ] <Question>

## Session log

### <YYYY-MM-DD HH:MM>
- What changed:
- Why:
- Blockers:
- Next:

## Commits
- <hash> - <message>

You can keep it simple. The win is consistency.

Session start and session end prompts

Start a session

Paste your phase prompt and current work notes, and tell the model to continue from the last session.

I'm continuing work on Phase <X>.

Prompt:
<paste prompts/phase-X.md>

Current state:
<paste work-notes/phase-X.md>

Please:
1. Summarize where we are (3 to 4 sentences).
2. List blockers and open questions.
3. Confirm the next logical unit.
4. Proceed with the next logical unit.

End a session

Before we stop:
1. Update the session log with what we did and what is next.
2. Ensure decisions, assumptions, and open questions are current.
3. Propose a commit message for any completed logical unit.
4. Show the updated work-notes file.

Verification

You can verify your notes are doing their job by forcing a cold start:

Start a new chat.
Paste only the phase prompt and the work-notes file.
See if you can resume without re-explaining anything.

Mechanical checks:

# Work notes exist and are non-empty.
find work-notes -type f -name '*.md' -maxdepth 2 -print -exec test -s {} \;

# Work notes have at least the core sections.
rg -n "^## (Status|Decisions|Assumptions|Open questions|Session log|Commits)" work-notes

Gotchas

Notes without rationale are not useful later.
If you let the model continue without updating notes, the next session will drift.
Avoid dumping raw logs with sensitive data. Sanitize first.

Continue -> Chapter 5: The Execution Loop: Review Discipline + Commit Discipline

Chapter 3: Prompt Documents: Prompts That Survive Sessions

Mon, 19 Jan 2026 12:00:00 -0500

Series: LLM Development Guide

Chapter 3 of 16

Previous: Chapter 2: Planning: Plan Artifacts, Constraints, Definition of Done

Next: Chapter 4: Work Notes: External Memory + Running Log

What you’ll be able to do

You’ll be able to create prompt documents that:

Encode intent precisely so you don’t re-explain yourself.
Align to plan phases so scope stays tight.
Include verification steps and explicit stop points.
Tell the model how to update work notes and propose commits.

TL;DR

A prompt doc is an artifact, not a chat message.
Use one prompt file per phase.
Always include: role, context, task, constraints, deliverables, verification, session management.
Put negative constraints in writing (“MUST NOT”).
Keep prompts copy/pasteable and path-specific.

Why prompt docs matter
Anatomy of a good prompt
A template you can copy
Verification
Failure modes

Why prompt docs matter

If you’re doing anything bigger than a one-off snippet, the prompt itself becomes part of the system.

Prompt docs:

Reduce “prompt drift” across sessions.
Make handoffs possible.
Create an audit trail of what was asked.
Force you to pin down deliverables and done-ness.

Anatomy of a good prompt

At minimum, include these sections:

Role: what expertise you’re invoking.
Context: plan path, work-notes path, reference implementation paths.
Task: what to do now.
Constraints: what must and must not happen.
Deliverables: exact files and outputs expected.
Verification: commands and expected results.
Session management: how to update work notes.
Commit discipline: atomic commits, propose messages, wait for approval.

If your prompt is missing verification and stop rules, you’re inviting “looks right” output.

A template you can copy

# Phase <X> - <Phase Name>

## Role
You are a senior software engineer.

## Context
- Plan: plan/<phase>.md
- Work notes: work-notes/<phase>.md
- Reference implementations:
 - <path 1>
 - <path 2>

## Task
Implement the next logical unit for this phase.

## Constraints (follow exactly)
- MUST follow patterns in the reference implementations.
- MUST keep changes scoped to this phase.
- MUST include tests when applicable.
- MUST propose verification commands.
- MUST NOT add new dependencies unless explicitly approved.

## Deliverables
1. <file path> - <what it contains>
2. <file path> - <what it contains>

## Session management
As you work:
- Update work-notes/<phase>.md:
 - Decisions (with rationale)
 - Assumptions
 - Open questions
 - Session log entry
- After each logical unit, pause and show the updated notes section.

## Verification
After implementing the logical unit, run or propose:
- <command>
- Expected: <exit 0 / output contains X>

## Commit discipline
After verification:
1. Summarize what changed and why.
2. Propose a conventional commit message.
3. Wait for approval before continuing.

Refs: work-notes/<phase>.md

Notes:

Use real paths.
Put constraints in a dedicated section.
Repeat the most important constraints near the end.

Verification

You can verify prompt docs are usable by checking two things:

You can paste the entire file verbatim into a new session.
A new session produces the same behavior because paths and constraints are explicit.

Concrete checks:

# Prompts exist and are non-empty.
find prompts -type f -name '*.md' -maxdepth 2 -print -exec test -s {} \;

# Prompts mention work-notes paths.
rg -n "work-notes/" prompts

Expected results:

Each prompt file is non-empty.
Each prompt references a work-notes file.

Failure modes

Prompts that say “use the config file” without a path.
Constraints buried in prose instead of a dedicated section.
Prompts that do not mention verification.
Prompts that do not tell the model to stop after a logical unit.

Continue -> Chapter 4: Work Notes: External Memory + Running Log

Chapter 2: Planning: Plan Artifacts, Constraints, Definition of Done

Sat, 17 Jan 2026 11:00:00 -0500

Series: LLM Development Guide

Chapter 2 of 16

Previous: Chapter 1: A Practical Workflow for LLM-Assisted Development That Doesn’t Collapse After Day 2

Next: Chapter 3: Prompt Documents: Prompts That Survive Sessions

What you’ll be able to do

You’ll be able to write a plan artifact that:

Forces clarity on scope, constraints, and references.
Produces verification steps (not just a task list).
Is sized so an LLM can execute it phase-by-phase without drifting.

TL;DR

A plan is a shared source of truth between you and the model.
Keep plans at the “what” level; keep “how” in prompt docs.
Every phase needs verification and a definition of done.
If a plan file would exceed ~200 lines, split it.
Always point to reference implementations by path.

What belongs in a plan (and what doesn’t)
A plan template you can paste
Sizing rules
Verification and definition of done
Verification
Gotchas

What belongs in a plan (and what doesn’t)

Plans work when they are explicit and boring.

Include:

Goals and non-goals.
Constraints and invariants.
Reference implementations (by path).
Phases in dependency order.
Verification for each phase.

Avoid:

Full code blocks.
Deep implementation detail.
“Make it better” language.

If you want the LLM to do a thing consistently across sessions, you need the thing written down.

A plan template you can paste

Create one plan file per phase for larger work.

# <Project> Plan

## Overview
<1 to 2 sentences about what we are building>

## Goals
- <Goal 1>
- <Goal 2>

## Non-goals
- <Explicitly out of scope>

## Constraints
- <Must follow reference style X>
- <Must not add dependencies>
- <Must keep backward compatibility>

## References
- <Path to reference implementation 1>
- <Path to reference implementation 2>

## Phase 1: <Name>
- [ ] <Task 1>
- [ ] <Task 2>

Verification:
- <Command>
- Expected: <Exit 0 / output contains X>

## Phase 2: <Name>
- [ ] <Task 1>

Verification:
- <Command>
- Expected: <...>

## Definition of done
- [ ] <All phases verified>
- [ ] <Tests pass>
- [ ] <Docs updated as needed>
- [ ] <No TODOs left behind>

## Risks / open questions
- <Open question 1>
- <Risk 1>

Sizing rules

You need the plan sized so the LLM can execute it without mixing unrelated changes.

Use these rules of thumb:

Small (hours to 1 to 2 days): one PLAN.md.
Medium (1 to 2 weeks): one PLAN.md with explicit phases.
Large (multi-week): plan/phase-1a-...md, plan/phase-1b-...md, etc.

When in doubt:

Split by file ownership (phases should avoid editing the same files).
Split by interface boundaries (one phase defines types/contracts; later phases implement).

Verification and definition of done

Make verification explicit in the plan so you don’t have to negotiate it mid-session.

Bad:

“Add tests”

Better:

“Add unit tests for Foo and run go test ./... (expected: exit 0).”

If your phase can’t be verified, it probably isn’t a phase yet.

Verification

If you follow the template above, you should be able to run something like:

# Example: lint and test gates.
# Replace with your repo's actual commands.

go test ./...

git diff --stat

Expected results:

go test exits 0.
git diff --stat shows only the files you intended to touch in this phase.

Gotchas

Plans that mix “what” and “how” become unreadable quickly.
If you don’t write down constraints, the LLM will invent defaults.
A “phase” that touches 30 files is usually multiple phases.

Continue -> Chapter 3: Prompt Documents: Prompts That Survive Sessions

Chapter 1: A Practical Workflow for LLM-Assisted Development That Doesn't Collapse After Day 2

Thu, 15 Jan 2026 09:00:00 -0500

Series: LLM Development Guide

Chapter 1 of 16

Next: Chapter 2: Planning: Plan Artifacts, Constraints, Definition of Done

What you’ll be able to do

You’ll be able to take a real development task and run an LLM through a repeatable loop that:

Survives breaks and multi-day work.
Produces output you can actually review.
Includes verification steps, not just code.
Creates a paper trail of decisions and assumptions.

TL;DR

Treat the LLM like a senior engineer who can execute quickly, but has no durable memory.
Externalize memory into three artifacts: plan/, prompts/, and work-notes/.
For large projects, add phase specification docs and phase implementation prompt docs (see Chapter 7).
Execute in small logical units, with verification and atomic commits.
If you’re fighting output quality, upgrade the model or shrink the scope.
Never paste secrets, PII, or production data.

Trust contract (read this before you paste anything)

Security: do not paste secrets, tokens, customer data, or anything you would not publish in a public repo.
Staleness: model names, pricing, and vendor policies change frequently. Treat examples as illustrative as of 2026-02-14.
Prereqs: you can run tests, review diffs, and explain the change in a code review.

Why most LLM-assisted development fails
The workflow
Quick start: copy/paste kit
Worked example: Helm chart from a reference chart
Verification
Failure modes

Why most LLM-assisted development fails

Most failures are workflow failures, not “prompting” failures:

You jump straight to implementation without a plan.
You don’t provide reference implementations, so you get generic output.
You lose context across sessions.
You don’t verify output.
You batch changes into giant commits that are hard to review or revert.

The workflow

This is the smallest loop I’ve found that stays stable after day 2:

Plan -> Prompt docs -> Work notes -> Execute -> Verify -> Commit

The artifacts are simple:

plan/: what we’re doing and how we’ll know it’s done.
prompts/: the reusable prompts aligned to phases.
work-notes/: state, decisions, assumptions, open questions, and a running session log.

When work scales to multi-week delivery, promote this into explicit phase specification docs plus phase implementation prompt files so scope and verification stay deterministic across sessions.

Quick start: copy/paste kit

This is intentionally minimal. It’s enough to make sessions resumable.

1) Create the artifact directories

mkdir -p plan prompts work-notes

2) Create a minimal plan

cat > plan/phase-1.md <<'MD'
# Phase 1: Plan

## Overview
<One sentence: what we are building>

## Goals
- <Goal 1>
- <Goal 2>

## Constraints
- <Constraint 1>
- <Constraint 2>

## Definition of done
- [ ] <Verification command + expected outcome>
- [ ] <Verification command + expected outcome>

## Out of scope
- <Thing we will not do in this phase>
MD

3) Create a phase prompt doc

cat > prompts/phase-1.md <<'MD'
# Phase 1 - Execution Prompt

## Role
You are a senior software engineer.

## Context
- Plan: plan/phase-1.md
- Work notes: work-notes/phase-1.md
- Reference implementation(s): <paths>

## Task
Implement the smallest logical unit that moves this phase forward.

## Constraints (follow exactly)
- MUST follow patterns in the reference implementation.
- MUST propose verification commands.
- MUST NOT change files outside this phase scope.

## Session management
As you work, update work-notes/phase-1.md:
- Decisions (with rationale)
- Assumptions
- Open questions
- Session log entry

## Commit discipline
After each logical unit:
1. Stop and summarize what changed.
2. Propose a commit message.
3. Wait for approval.
MD

4) Create work notes

cat > work-notes/phase-1.md <<'MD'
# Phase 1 - Work Notes

## Status
- [ ] Not started
- [ ] In progress
- [ ] Blocked
- [ ] Complete

## Decisions

## Assumptions

## Open questions

## Session log

## Commits
MD

Worked example: Helm chart from a reference chart

This example is about correctness and maintainability, not “Helm tricks”.

Scenario

Goal: create a new chart (for example, metrics-gateway) by following a reference chart (for example, event-processor) that already works in your environment.

The important part is the inputs you give the model. Don’t describe the reference chart. Paste it.

Reference inputs (what to paste)

Run these commands in your repo and paste their output into your planning prompt:

tree charts/event-processor/
sed -n '1,200p' charts/event-processor/Chart.yaml
sed -n '1,200p' charts/event-processor/values.yaml
sed -n '1,200p' charts/event-processor/templates/_helpers.tpl

Plan prompt (high-signal)

I want to create a new Helm chart for a service called `metrics-gateway`.

Reference implementation: charts/event-processor/ (this is our standard).
The new chart MUST follow the same structure and conventions.

Here are the reference inputs:
- tree output: ...
- Chart.yaml: ...
- values.yaml: ...
- templates/_helpers.tpl: ...

Please:
- Analyze the reference chart patterns.
- Produce a phased plan with verification steps.
- Call out any open questions you need answered (ports, probes, resources).

Execution prompt (phase-aligned)

Once you have the plan, generate prompt docs aligned to phases (scaffold, core templates, env overrides, validation). Each prompt should:

Name the deliverables.
Repeat constraints.
Include “update work notes” instructions.
Include verification commands.

What “done” looks like

A good end state is boring:

The new chart is structurally identical to the reference chart.
The values structure matches (so operators don’t re-learn config surfaces).
helm lint and helm template succeed.
Changes are split into reviewable commits.

Verification

You can verify you’re actually following the workflow, not just producing text:

# Artifacts exist.
test -d plan && test -d prompts && test -d work-notes

# A plan exists and is not empty.
test -s plan/phase-1.md

# A prompt doc exists.
test -s prompts/phase-1.md

# Work notes exist.
test -s work-notes/phase-1.md

If you are doing the Helm chart example:

helm lint charts/metrics-gateway
helm template charts/metrics-gateway >/tmp/metrics-gateway.rendered.yaml
test -s /tmp/metrics-gateway.rendered.yaml

Expected results:

The commands exit with code 0.
The rendered YAML file is non-empty.

Failure modes

Skipping references: you get generic output that doesn’t match your repo.
Skipping verification: you ship code that “looked right.”
Letting sessions run too long: context drifts and you lose earlier constraints.
Batching commits: review slows down and rollback gets painful.
Using the wrong model: cheap models are fine for boilerplate, but can burn hours on complex reasoning.

Continue -> Chapter 2: Planning: Plan Artifacts, Constraints, Definition of Done

Agile Isn't Dead. Agile Compliance Is.

Wed, 31 Dec 2025 12:00:00 -0500

Note on examples: The scenarios below are anonymized composites. This isn’t “Agile bad.” It’s “Agile the brand is often used to justify systems that do the opposite of Agile’s intent.”

Why this matters

Agile isn’t a set of meetings. It’s a physics statement:

Shorter feedback loops reduce risk.

Most enterprises didn’t fail Agile. They replaced Agile with a bureaucracy that uses Agile vocabulary:

“Sprint” becomes a reporting interval
“Velocity” becomes a performance metric
“Planning” becomes a negotiation
“Definition of done” becomes a checklist
“Agile transformation” becomes a multi-year program

The result is predictable:

delivery slows
quality degrades
reliability suffers
engineers burn out
product expectations aren’t met
leadership gets more dashboards and fewer outcomes

This post is a production-first teardown of Agile theater - and a replacement model that actually ships.

TL;DR

Agile is about learning quickly, not predicting perfectly.
Scrum is useful when it reduces uncertainty. It’s harmful when it becomes a compliance system.
If you treat sprints as contracts, you’ll get scrumfall: waterfall dependencies with sprint-shaped reporting.
Replace “Agile compliance” with:
Flow (small batches, limit WIP)
Continuous delivery (safe, frequent releases) [4]
Evidence-based planning (measure outcomes; adjust quickly) [5]
Use system metrics (DORA) to verify improvement: lead time, deploy frequency, change failure rate, MTTR. [6]
Beware Goodhart’s Law: metrics used as targets will be gamed. [7]

Agile the physics vs Agile the bureaucracy
Pattern 1: Sprints as contracts
Pattern 2: Velocity as a performance metric
Pattern 3: Backlog bloat as a museum of anxiety
Pattern 4: Ceremonies become the work
Pattern 5: Dependencies turn Scrum into fiction
Pattern 6: Definition of done without production
Pattern 7: Product ownership by proxy
What’s better: Flow + CD + evidence
Transition plan: 30 days without a revolution
Verification: how you know it’s working
A practical checklist
References

Agile the physics vs Agile the bureaucracy

The Agile Manifesto values working software over comprehensive documentation and emphasizes collaboration and responding to change. [1] One of its principles states that working software is the primary measure of progress. [2]

Those ideas are still correct.

What broke in enterprises is implementation:

Agile became process instead of feedback
agile artifacts became deliverables
teams were optimized for predictability theater instead of throughput and learning

In short: Agile got turned into compliance.

Pattern 1: Sprints as contracts

What it looks like

Sprint planning is treated as a commitment contract.
Changing scope is seen as failure, even when reality changes.
Teams avoid surfacing unknowns because unknowns disrupt “commitment.”

Why it happens

Leaders want predictability. Sprints feel like a way to buy it.

The hidden tax

When you turn sprints into contracts, teams adapt:

reduce exploration
defer integration
accept low-quality shortcuts
split work into artificial “done-looking” chunks

You don’t eliminate uncertainty. You hide it until the end.

The replacement pattern

Use cadence as a heartbeat, not as a contract:

Plan in small chunks.
Commit to outcomes and constraints, not a stack of tickets.
Treat scope as a lever; treat time as a constraint.

Pattern 2: Velocity as a performance metric

What it looks like

Story points become productivity.
Velocity is compared across teams.
Teams feel pressure to “go faster” by increasing points delivered.

Why it happens

Velocity is a number. Numbers are tempting.

The hidden tax

Story points are a local measure with no consistent meaning across teams. When you attach incentives, teams optimize for the metric:

inflate estimates
split work to maximize points
avoid hard, high-leverage work
ship low-value changes

This is a textbook Goodhart’s Law failure mode: when a measure becomes a target, it ceases to be a good measure. [7]

The replacement pattern

Measure the system, not the story:

lead time
cycle time
deploy frequency
change failure rate
MTTR

Use metrics diagnostically, not as quarterly targets.

Pattern 3: Backlog bloat as a museum of anxiety

What it looks like

Thousands of backlog items exist “for visibility.”
Nothing gets deleted.
Refinement happens continuously, but priorities change weekly.

Why it happens

Backlogs feel like control: “We haven’t forgotten.”

The hidden tax

A giant backlog increases planning cost and reduces focus. Teams stop trusting priorities and operate on side-channel requests.

My favorite framing:

If everything is in the backlog, nothing is prioritized. It’s just a museum of anxiety.

The replacement pattern

Adopt a tight horizon model:

Now: what we’re building
Next: what’s likely next
Later: ideas (low-investment capture)

Refine Now/Next. Archive the rest.

Pattern 4: Ceremonies become the work

What it looks like

Standups become status meetings for managers.
Planning takes hours.
Refinement is endless.
Retrospectives generate action items that never get resourced.

Why it happens

Ceremonies are easy to schedule. Delivery capability is harder to build.

The hidden tax

Attention becomes fragmented. Engineers become “meeting responders.” Work gets multi-tasked across initiatives.

This is how you get:

slow delivery
low quality
burnout

The replacement pattern

Keep only the meetings that reduce uncertainty:

shorter planning
true async refinement
standup for coordination within the team (not reporting)
retros with real ownership and budget

Then invest in the thing ceremonies can’t replace: engineering capability (tests, pipelines, observability, automation).

Pattern 5: Dependencies turn Scrum into fiction

What it looks like

Every story depends on another team.
“Blocked” is normal.
Integration is deferred to later sprints.

Why it happens

Organizations are siloed. Systems mirror communication structures (Conway’s Law). [8]

The hidden tax

You get scrumfall: waterfall dependencies, sprint-shaped reporting.

A two-week sprint can’t save a three-month dependency queue.

The replacement pattern

Design for end-to-end ownership and flow:

reduce handoffs
remove or automate cross-team gates
create platform paved roads so teams can self-serve [9]

When dependencies can’t be eliminated, make them explicit and manage them like risk, not like hope.

Pattern 6: Definition of done without production

What it looks like

“Done” means “merged.”
QA is a phase.
Observability is optional.
Releases happen “later.”

Why it happens

Shipping is painful. So teams avoid it.

The hidden tax

If “done” doesn’t include production, you accumulate:

integration debt
release debt
incident debt

Reliability declines because feedback arrives late.

Continuous delivery’s core argument is that keeping software deployable and releasing frequently reduces risk and enables faster feedback. [4]

The replacement pattern

Upgrade your definition of done:

deployed to a real environment
observable (metrics/logs/traces)
rollback path exists
runbook exists for major failure modes

Pattern 7: Product ownership by proxy

What it looks like

Engineers rarely talk to users/operators.
“Product” is a chain of intermediaries.
Requirements arrive as polished tickets without the “why.”

Why it happens

The organization tries to protect engineers from churn.

The hidden tax

This degrades the input signal. Engineers build the wrong thing efficiently - and then everyone is surprised it didn’t land.

The replacement pattern

Bring engineers closer to reality:

listen to customer calls
review usage telemetry
participate in discovery
keep the “why” attached to every build

No one should ship something they can’t explain.

What’s better: Flow + CD + evidence

If Agile compliance is the disease, what’s the cure?

It’s not “a different framework.” It’s an operating model:

1) Flow: small batches, limited WIP

Lean/Kanban concepts focus on limiting work in progress and optimizing for flow. [3]

Finish work, don’t start work.
Reduce batch size.
Make queues visible.

2) Continuous Delivery: make change safe

Continuous delivery is a capability: keep changes small, deployable, and observable so you can release frequently with lower risk. [4]

This includes:

CI
automated testing
progressive delivery (when needed)
rollback/roll-forward discipline
telemetry tied to releases

3) Evidence-based planning: bets, not contracts

Lean Startup’s build-measure-learn loop emphasizes validated learning - ship something real, measure, and adjust. [5]

For enterprises, the translation is simple:

Plan in small bets
Validate early
Use evidence to re-plan, not politics

Transition plan: 30 days without a revolution

You don’t need to burn the framework down. You need to change what you reward and what you ship.

Week 1: Make work visible as flow

Map the value stream from idea -> production.
Count handoffs.
Measure current lead time.

Week 2: Reduce batch size

Pick one initiative.
Cut it to a thin vertical slice that can ship.
Define “done” as “in production, measurable.”

Week 3: Reduce WIP

Stop starting new work.
Finish the slice.
Remove one blocking dependency with a paved path or automation.

Week 4: Close the feedback loop

Ship.
Measure.
Run a retro focused on system constraints (not blame).
Repeat.

If you do this and nothing improves, you learned something valuable: the constraint is elsewhere.

Verification: how you know it’s working

You should see movement in system outcomes:

DORA describes four key delivery performance metrics: lead time for changes, deployment frequency, change failure rate, and time to restore service. [6]

Signs of real improvement:

lead time drops (less queueing and fewer handoffs)
deploy frequency rises (smaller batches, calmer releases)
change failure rate drops (better tests and safer rollouts)
MTTR drops (better observability and operability)

And importantly: teams report less “deployment pain” and less burnout as delivery becomes calmer and more reliable. [10]

A practical checklist

If you’re stuck in Agile theater, try this:

Stop measuring activity

Stop comparing velocity across teams.
Stop treating story points as productivity.

Shrink feedback loops

Ship a thin slice to production early (behind a flag if needed).
Put engineers closer to users/operators.

Reduce handoffs and WIP

Limit concurrent initiatives.
Remove one handoff per quarter.

Invest in delivery capability

CI, tests, deployment automation
observability tied to releases
safer rollouts and rollback paths

Use metrics as signals, not targets

Track DORA metrics at the system level. [6]
Avoid metric gaming (Goodhart). [7]

References

[1] Manifesto for Agile Software Development (values). https://agilemanifesto.org/ [2] Principles behind the Agile Manifesto (“Working software is the primary measure of progress”). https://agilemanifesto.org/principles.html [3] Kanban Guide (principles and practices oriented around flow and WIP). https://kanbanguides.org/english/ [4] Continuous Delivery (concepts; keep software deployable, release frequently). https://continuousdelivery.com/ [5] The Lean Startup - Principles (Build-Measure-Learn; validated learning). https://theleanstartup.com/principles [6] DORA - “DORA’s software delivery performance metrics (guide)”. https://dora.dev/guides/dora-metrics/ [7] CNA - “Goodhart’s Law” (when a measure becomes a target, it ceases to be a good measure). https://www.cna.org/analyses/2022/09/goodharts-law [8] Splunk - “Conway’s Law Explained” (systems mirror communication structures; includes original quote). https://www.splunk.com/en_us/blog/learn/conways-law.html [9] Microsoft Engineering Blog - “Building paved paths: the journey to platform engineering”. https://devblogs.microsoft.com/engineering-at-microsoft/building-paved-paths-the-journey-to-platform-engineering/ [10] DORA - “Capabilities: Well-being” (deployment pain and relationship to performance/culture). https://dora.dev/capabilities/well-being/

Agent Observability That Doesn't Lie

Sat, 20 Dec 2025 12:00:00 -0500

Why this matters

Most “agent observability” is either:

too shallow (a chat transcript and a couple logs), or
too noisy (every token logged, every tool payload stored, no signal)

Neither works in production.

If you’re serious about operating agents, you need observability that answers three questions quickly:

What happened? (forensics)
Why did it happen? (debuggability)
How often does it happen? (reliability)

OpenTelemetry exists to standardize how you instrument, generate, and export telemetry across traces, metrics, and logs. [1] W3C Trace Context defines how trace context propagates across service boundaries. [2]

Agents add two new requirements:

tool calls are part of your “distributed trace”
“decisioning” is a first-class component (not just business logic)

This article is a practical blueprint.

TL;DR

Instrument agents like distributed systems:
traces for causality (what triggered what)
metrics for health (p95 latency, error rates)
logs for human context (but redacted)
Propagate a single trace across:
agent runtime -> MCP gateway -> MCP tool servers -> upstream APIs
Capture decision summaries, not chain-of-thought.
Treat cost as a production signal: emit per-run and per-tool cost metrics.
Use semantic conventions where possible to keep telemetry queryable. [3]
Don’t turn observability into a data breach: OWASP highlights sensitive info disclosure and prompt injection as key risks. [7]

What to observe in an agent system
A trace model for agents
Metrics that matter
Logs and redaction
Audit events vs debug logs
Dashboards and alerts
A production checklist
References

What to observe in an agent system

Agents have four observable subsystems:

Planner/Reasoner (creates the plan, chooses tools)
Tool execution (calls MCP tools and interprets results)
Memory/state (what was stored or retrieved)
Policy/budget (what was allowed or blocked)

If you only observe #2, you’ll miss why the agent chose the wrong tool. If you only observe #1, you’ll miss production failures.

You need the full chain.

A trace model for agents

The core idea

A single “agent run” is a distributed trace:

it spans model calls
tool calls
downstream system calls

Use W3C Trace Context (traceparent, tracestate) to propagate the trace across boundaries. [2]

Suggested spans (minimum viable)

Root span

agent.run
attributes: agent.name, tenant, user, session, goal_hash

Planner

agent.plan
attributes: planner.model, plan.step_count

Model calls

llm.call
attributes: model, prompt_tokens, completion_tokens, latency_ms

Tool selection

agent.tool_select
attributes: selector.version, candidate_count, selected_count

Tool call

tool.call
attributes: tool.name, tool.class (read/write/danger), tool.server, status

Policy

policy.check
attributes: policy.rule_id, decision (allow/deny), reason_code

Memory

memory.read / memory.write
attributes: store, keys, bytes

Why spans > logs

Spans give you causality:

which tool call caused a failure
which step blew the budget
which upstream dependency was slow

With OpenTelemetry, you can emit traces and metrics using the same SDK approach. [1][4]

Metrics that matter

Tool health metrics

tool_calls_total{tool,status}
tool_latency_ms_bucket{tool}
tool_timeouts_total{tool}
tool_retries_total{tool}

Agent run health metrics

agent_runs_total{status}
agent_run_latency_ms_bucket{agent}
agent_steps_total_bucket{agent}

Cost metrics (treat cost like reliability)

llm_tokens_total{model,type=prompt|completion}
llm_cost_usd_total{model}
run_cost_usd_bucket{agent}

Policy metrics

policy_denied_total{rule_id}
danger_tool_attempt_total{tool}

Semantic conventions help your metrics stay queryable and consistent across systems. OpenTelemetry documents semantic conventions for HTTP spans/metrics, for example. [3][5]

Logs and redaction

Logs should add human context, not become a data lake of secrets.

Rules I like:

Do not log prompts by default.
Do not log tool payloads by default.
Log summaries and hashes:
goal_hash, plan_hash, tool_args_hash
Log structured error reasons:
validation_error, upstream_rate_limited, auth_failed, policy_denied

For agent systems, OWASP highlights sensitive information disclosure and insecure output handling. Logging is one of the easiest ways to accidentally create both. [7]

“Debug mode” that isn’t dangerous

If you must support deeper logs:

only enable per tenant/user for a limited window
auto-expire
redact aggressively
never store raw secrets

Audit events vs debug logs

Treat them as different products:

Audit events (for governance)

immutable-ish records of side effects
minimal sensitive data
always on
long retention

Example audit fields:

who: tenant/user/client
what: tool + action class (create/update/delete)
when: timestamp
where: environment
result: success/failure
resource IDs (safe identifiers)
idempotency keys / plan IDs

Debug logs (for engineers)

short retention
more context
highly controlled access

Mixing these two is how you end up with “SharePoint logs full of PII” and no one wants to touch them.

Dashboards and alerts

Dashboards (start simple)

Tool reliability

top tools by error rate
top tools by p95 latency
timeouts per tool

Agent success

success rate by agent type
“stuck runs” (runs exceeding max duration)
average steps per run

Cost

cost per run
cost per tenant
top drivers (which tools/model calls)

Alerts (avoid noise)

Alert on what is actionable:

tool error rate spikes for critical tools
tool latency p95 spikes beyond SLO
budget exceeded spike (runaway behavior)
policy denied spike (possible prompt injection attempt)

If you use SLOs and error budgets, Google’s SRE material is a practical reference for turning SLOs into alerting strategies. [6]

A production checklist

Tracing

Every agent run has a trace ID.
Trace context propagates across MCP boundaries (W3C Trace Context). [2]
Tool calls are spans with stable tool identifiers.

Metrics

Tool success/error/latency metrics exist.
Agent run success/latency/steps metrics exist.
Cost metrics exist and are monitored.

Logging

Default logs are redacted summaries, not raw payloads.
Debug logging is time-bounded and access-controlled.

Audit

Audit events exist for all side-effecting tools.
Audit records include “who/what/when/result” without leaking secrets.

Security

Observability does not become a secret exfil path (OWASP risks considered). [7]

References

[1] OpenTelemetry - Documentation (overview): https://opentelemetry.io/docs/ [2] W3C - Trace Context: https://www.w3.org/TR/trace-context/ [3] OpenTelemetry - Semantic conventions for HTTP (spans/metrics/logs): https://opentelemetry.io/docs/specs/semconv/http/ [4] OpenTelemetry Go - Instrumentation docs: https://opentelemetry.io/docs/languages/go/instrumentation/ [5] OpenTelemetry - Semantic conventions for HTTP metrics: https://opentelemetry.io/docs/specs/semconv/http/http-metrics/ [6] Google SRE Workbook - Alerting on SLOs: https://sre.google/workbook/alerting-on-slos/ [7] OWASP - Top 10 for Large Language Model Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/

Cost Is a Reliability Problem

Sat, 13 Dec 2025 12:00:00 -0500

Why this matters

Traditional reliability focuses on uptime. AI systems add a second axis:

Your system can be “up” while your budget is on fire.

A runaway agent doesn’t always crash services. Sometimes it:

loops tool calls
retries incorrectly
escalates to larger models repeatedly
expands context windows unnecessarily
performs expensive searches without stopping

The result: surprise bills, throttling, and eventually hard outages when quotas are hit.

Google’s SRE framing around error budgets is a useful mental model: budgets create a control mechanism that balances stability with velocity. [1][2] FinOps frames cost management as a collaboration practice between engineering, finance, and business. [3]

This article is the practical bridge: use budgets and guardrails like you would for reliability.

TL;DR

Treat cost as an SLO: define acceptable spend per run / per tenant / per day.
Enforce budgets at multiple layers:
per request/run
per tool
per tenant
per environment
Use hard limits + soft limits:
soft: degrade model/tool choices
hard: stop the run and ask for approval
Add cost circuit breakers:
abort on runaway loops
quarantine tools causing repeated retries
Make cost visible (metrics + dashboards) so teams can improve it.
Align with FinOps: shared accountability, not “billing surprises.” [3]

Cost failure modes in agent systems
Define cost SLOs and budgets
Budget layers: run, tool, tenant, environment
Soft limits vs hard limits
Circuit breakers for runaway behavior
Cost-aware tool and model selection
Dashboards and alerts
A production checklist
References

Cost failure modes in agent systems

1) Infinite or long loops

Common triggers:

ambiguous tool outputs
brittle parsing
“try again” reflexes
non-idempotent retries

2) Tool spam

Agents sometimes “search until confident.” If you don’t cap it, you get 20+ tool calls on a single request.

3) Model escalation cascades

If your policy says “if uncertain, use a better model,” you can create a cost escalator:

cheap model -> “uncertain” -> expensive model
expensive model -> still uncertain -> more calls

4) Context growth

If you keep appending tool outputs to the prompt, costs grow superlinearly and performance can degrade.

5) External quotas become outages

Even if cost is acceptable, external services (email APIs, GitHub, calendars) can rate limit you. Cost and reliability are coupled.

Define cost SLOs and budgets

Start with simple “production truths”:

How much is one agent run allowed to cost?
What is an acceptable daily spend per tenant?
What is the max “blast radius” of a single request?

This maps cleanly to SRE’s error budget concept: budgets constrain unsafe behavior while preserving velocity. [2]

Example cost SLOs (pragmatic)

Per run: <= $0.10 (p95), <= $0.50 (max)
Per tenant/day: <= $50/day
Per user/day: <= $5/day
Per tool call: <= 3 calls to expensive tools

These aren’t universal. They’re explicit. That’s what matters.

Budget layers: run, tool, tenant, environment

1) Per-run budget

Tracks:

max model tokens
max tool calls
max wall-clock time
max “expensive operations” count

Most important budget. This is where you stop runaway behavior early.

2) Per-tool budget

Some tools are inherently expensive:

large searches
long-running jobs
heavy data exports

Budget these separately:

max calls
max payload size
max time range

3) Per-tenant budget

Without this, your best customers can melt your infra.

Per-tenant limits:

requests/min
concurrent runs
daily cost cap

4) Per-environment budget

Environments have different rules:

dev: cheap, permissive, more logging
prod: bounded, gated, auditable

This is where you implement “read-only mode” during incidents.

Soft limits vs hard limits

Soft limits (degrade gracefully)

When approaching budget:

switch to cheaper models
reduce context size (summarize)
narrow tool search range
skip non-essential steps

Hard limits (stop the run)

When budget is exceeded:

stop tool calls
stop escalation
request user confirmation / approval
produce a partial answer with an explanation

This is exactly the “control mechanism” idea behind error budgets: it gives the system permission to shift focus when constraints are exceeded. [1]

Circuit breakers for runaway behavior

Add circuit breakers that detect “this is going bad”:

loop detector: same tool called with similar args repeatedly
retry storm: high retry count for a tool within a run
no progress: plan step count increases without new evidence
latency breaker: tool p95 spikes beyond threshold

When triggered:

stop the run
quarantine the tool for this run
degrade to safe alternatives
emit high-signal telemetry

Cost-aware tool and model selection

Cost control is easier if it’s designed into selection:

Rank tools with a “cost weight” (latency + upstream cost + risk)
Prefer read-only tools unless a write is required
Use caches for common retrieval results
Use deterministic summarization boundaries for tool outputs

If you already implement a tool selector (see “Million Tool Problem”), cost becomes another rerank feature.

Dashboards and alerts

This is where FinOps and SRE meet: cost is an operational signal.

Dashboards

spend/day by tenant
cost per run distribution
top cost drivers (tools and models)
runaway breaker triggers

Alerts

daily spend exceeded
sudden spend spikes (slope alerts)
high frequency of loop breaker events
high fraction of runs hitting hard limits

AWS’s Well-Architected Cost Optimization pillar frames cost optimization as a continual process across the workload lifecycle. That mindset applies here too. [4]

A production checklist

Budgets

Per-run cost and tool-call budgets exist.
Per-tenant daily caps exist.
Per-tool “expensive operation” caps exist.

Enforcement

Soft limits degrade gracefully (cheaper models, narrower queries).
Hard limits stop and request approval.
Circuit breakers detect loops/retry storms.

Telemetry

Cost metrics emitted per run and per tenant.
Breaker events recorded and alertable.

Culture

Cost management is a shared practice (FinOps), not a surprise invoice. [3]

References

[1] Google SRE Workbook - Example Error Budget Policy: https://sre.google/workbook/error-budget-policy/ [2] Google SRE Book - Embracing Risk (error budgets as control mechanism): https://sre.google/sre-book/embracing-risk/ [3] FinOps Foundation - What is FinOps? (definition and principles): https://www.finops.org/introduction/what-is-finops/ [4] AWS Well-Architected Framework - Cost Optimization pillar: https://docs.aws.amazon.com/wellarchitected/latest/framework/cost-optimization.html

Roy Gabriel - DevOps Architect & Applied AI Engineer | Roy Gabriel

Agent Swarms: The New Workforce Architects and Engineers Must Lead

Why this shift keeps showing up

TL;DR

Contents

What Agent Swarms Do Today

Cruvero as the Working Example

The New Role for Architects and Engineers

Why You Must Start Today

References

Chapter 16: Worked Example: Converting an Ansible Playbook to a Go Temporal Workflow

What you’ll be able to do

TL;DR

Table of contents

Scenario

Reference inputs

Plan and phase structure

Implementation skeleton (Go)

Verification

Gotchas

Chapter 15: Worked Example: Creating a Helm Chart From a Reference Chart

What you’ll be able to do

TL;DR

Table of contents

Scenario

Reference inputs

Phase 1: Plan

Phase 2: Prompt docs

Phase 3: Execute in logical units

Strategy A (recommended): copy the reference chart, then adapt

Strategy B: scaffold from scratch, guided by the reference

Verification

Gotchas

Chapter 14: Building a Prompt Library: Governance + Quality Bar

What you’ll be able to do

TL;DR

Table of contents

Library structure

Prompt entry template

Contribution guidelines

Governance

Verification

When Enterprise Defaults Become Enterprise Debt

Why this matters

TL;DR

Contents

Pattern 1: Analysis as a substitute for delivery

What it looks like

Why it existed

The hidden tax

The replacement pattern

Transition step (low drama)

Pattern 2: Reinventing commodity infrastructure

What it looks like

Why it existed

The hidden tax

The replacement pattern

Transition step

Pattern 3: VM-first thinking as the default

What it looks like

Why it existed

The hidden tax

The replacement pattern

Transition step

Pattern 4: Ticket-driven infrastructure

What it looks like

Why it existed

The hidden tax

The replacement pattern

Transition step

Pattern 5: Change Advisory Board for routine changes

What it looks like

Why it existed

The hidden tax

The replacement pattern

Transition step

Pattern 6: The shared database empire

What it looks like

Why it existed

The hidden tax