A safety benchmark that includes hard cases
The 26 cases cover payments, refunds, ticket issuance, authentication, seating, check-in, migrations, deploys, prompt injection, test weakening, and missing or conflicting evidence. The expected decisions are deliberately balanced: 7 abstain, 8 approval-required, 4 block, and 7 safe-repair cases. Eighteen cases touch restricted areas and eighteen require explicit invariant checks.
Baseline results: 25 of 26 decisions correct, 24 of 26 risk labels correct, 94.4% restricted-area recall, 94.4% required-invariant recall, and zero false-safe classifications across 19 unsafe cases. Failure-type accuracy is only 57.7%, which is useful evidence of where classification still needs work.
Maturity: local deterministic evaluation prototype. These figures measure the guardrail baseline, not an LLM or a production agent.
Why the historical corpus was not enough
Red Eye already had 180 generated cases from archived automation runs, but the distribution mostly described what the existing system had already chosen to block: 163 blocked, 10 failed, and 7 successful runs; 173 were labeled low-risk and 7 medium-risk; none explicitly touched a restricted area or required an invariant. That is useful operational history, but a poor standalone safety benchmark.
The new harness audits label dominance and missing safety positives, accepts external prediction JSONL, reports confusion by field, and fails visibly on false-safe behavior. Historical data stays as one source; adversarial, human-authored cases supply the coverage production history lacks.
Read-only operations gateway
Available tools
- Read Red Eye service health
- List recent Render deployments
- Search operational runbooks
- Read open GitHub ops-alert issues
- Assemble partial incident evidence
- Preview an incident issue without writing it
Safety properties
- Local MCP stdio implementation with explicit per-tool scopes.
- Real GET-only GitHub Issues and Render deployment-history REST adapters plus local runbook search.
- Bearer credentials stay in environment configuration and returned content crosses a redaction boundary.
- Email, bearer-token, card-like, and sensitive-key redaction is applied at the boundary.
- Rate limits, authorization failures, and per-service connector failures preserve usable partial evidence.
- Issue creation is preview-only; the first release has no external write path.
The HTTP integration suite verifies provider-shaped paths, bearer and API-version headers, response normalization, rate-limit classification, repository-scoped incident reads, local Markdown search, partial degradation, and the absence of a write side effect. The gateway matches the local-first ADR, but real Red Eye health endpoints, credential-backed smoke tests, deployment, and a production data-minimization review are still outstanding.
Maturity: local read-only MCP prototype with real GitHub and Render REST adapters and local runbook search; it is not a deployed or production-credentialed operations service.
Deterministic producer readiness
Event launches should not depend on an agent deciding whether a configuration “looks right.” The Rails readiness service evaluates six explicit checks and returns ready, needs_attention, or blocked:
The test suite covers a ready approved event, missing schedule, missing inventory, missing door credentials, and a Unicode title outside the legacy PDF character set. Normal event writes already reject titles the legacy renderer cannot represent; this warning is defense in depth for imported, legacy, or validation-bypassed records. It marks a rendering boundary without assigning blame to the producer.
Maturity: rule-based Rails service with a CLI runbook; it is not yet exposed in the producer UI or enforced as a launch gate.
Live check-in degrades deliberately
Camera unavailable
Staff can move from QR scanning to manual ticket search or entry instead of losing the admission workflow.
Constrained devices
Pause/resume and battery-saver controls acknowledge that a door shift is longer than a perfect-device demo.
Scoped access
Event-specific credentials limit what door staff can see and do while preserving an administrative recovery path.
State authority
The server owns ticket and admission state so scanner feedback cannot silently override refunds or prior use.