Red Eye Tickets

Production implementation

From signed evidence to verified recovery

A guarded AI-assisted operations control plane turns production signals into bounded repair work without transferring unrestricted production authority to an agent.

Implemented in Red EyeHuman approval for higher-risk lanesEvidence reviewed July 2026
01

The control flow

01

Signed evidence

Production reports, alerts, deployment state, and runbooks cross a redaction and corroboration boundary.

02

Bounded task

Findings are fingerprinted, deduplicated, clustered, risk-classified, and assigned tests and a deployment lane.

03

Separated passes

Planner, executor, and reviewer work in isolation; the reviewer evaluates the diff and verification evidence.

04

Policy and approval

Restricted paths, test integrity, diff size, secrets, risk, and human-approval requirements fail closed.

05

Exact-commit release

Approved work moves through the certified deployment path with a recorded rollback plan and release identity.

06

Verified recovery

The incident remains unresolved until post-deploy probes pass and required reconciliation or replay completes.

02

Why the stages stay separate

Planner

Investigates the evidence, names uncertainty, identifies invariants, defines the narrow change, and records a rollback path.

Executor

Works inside the permitted scope, preserves tests and restricted boundaries, and produces the implementation and verification artifacts.

Reviewer

Evaluates the diff, test evidence, risk classification, missing proof, and whether release criteria actually passed.

Release authority

Policy and human approval determine whether work may stage, merge, deploy, or stop. The agent does not silently widen its own authority.

# Sanitized excerpt from the production runner
planner_pass
executor_pass
reviewer_pass

# Deployment is incomplete until the explicit verifier succeeds.
run_production_verification || abort_blocked(
  "Production verification failed after deploy."
)

Excerpt boundary: representative names and control order are preserved; paths, environment values, credentials, identifiers, and security-sensitive implementation details are excluded.

03

Policy and approval are part of the implementation

Fail-closed checks

  • Restricted payment, authentication, database, workflow, and infrastructure paths.
  • Required invariant and regression coverage.
  • Diff-size, file-count, secret-redaction, and test-integrity limits.
  • Risk, severity, evidence quality, deploy lane, and approval requirements.

Release controls

  • Exact-commit deployment across the required production services.
  • Cross-worktree and deployment locks, cooldowns, release fences, and rollback plans.
  • Fresh production probes, canaries, recovery telemetry, and required replay or reconciliation.
  • Quiet success and loud uncertainty: ambiguous evidence escalates instead of being converted into a confident mutation.
04

Loop: observation to outcome

Loop is the internal name for a policy-governed observation and learning architecture. It connects production evidence to action while preserving the authority, causal, and maturity boundaries required for a financial and ticket-state system.

01Observation02Finding03Proposed action04Approval05Rollback plan06Verification07Outcome08Lesson

The outcome is not “the agent learned.” The system records which observation produced which finding, what action was proposed and approved, how rollback and verification were defined, what happened after release, and whether a reusable lesson is safe to promote.

05

Evidence from recent delivery

6,628Backend examples passed in production-canary and automation hardening
358Focused examples for the attended Loop acceptance contract
64Focused examples for independent Watchdog heartbeat monitoring
0Failures in the cited validation runs
  • Production-canary and automation hardening distinguished classification defects from downtime and preserved fail-closed checks.
  • An attended Loop acceptance contract captured immutable identity, bounded provider-window evidence, shutdown, and rollback behavior.
  • Independent Better Stack monitoring detects failure of the external Watchdog itself, adding a separate failure-domain signal.
  • GitHub evidence records risk, tests, rollout, backout, exact release identity, and post-deploy acceptance criteria.

Test-summary boundary: counts are taken from the cited merged PR validation records. They describe those runs; they are not a claim that every repository test belongs to this subsystem.

06

Maturity boundaries

Production-facing

Signed operational evidence, alert intake, task generation, policy enforcement, deployment controls, canaries, Watchdog monitoring, and production verification.

Local execution plane

Planner, executor, and reviewer passes run in isolated worktrees with full developer tooling and explicit release authority.

Attended or pilot-scoped

Loop observation proofs use reviewed activation windows, immutable contracts, bounded occurrences, shutdown, and retirement evidence.

Default-off or framework-only

Capabilities that are not authorized for ordinary production execution remain ineligible by configuration and policy.