Assess supervision quality

Evaluation reports record model settings, prompt, schema and case digests, timing, usage, outcomes, and evidence provenance. Ask your integration operator for the report and its evaluation conditions when assessing a workflow.

Scripted outputs validate contracts and enforcement. They do not measure a live model's reasoning or refusal quality. Frozen classifications do not establish end-to-end task completion.

Quality rates use completed eligible samples. Missing credentials, unavailable evidence, and provider errors are reported separately. Reports distinguish unnecessary refusals, missed refusals, inappropriate approvals, paraphrase agreement, task completion, recovery success, and unauthorized persisted effects. Some metrics do not apply to frozen judgments and have no denominator.

Compare like-for-like providers, model settings, case versions, and conditions. Check repetitions and sample counts before interpreting an apparent improvement. Live runs use bounded call budgets; incomplete or unavailable conditions must remain visible in the report.

Readable reasoning evidence is optional and untrusted. Live frozen comparisons capture provider evidence once per provider/case and require matching action digests. Missing summaries remain explicitly absent. Provider-exported summaries are not private chain of thought and do not prove how a model reached its decision.

Keep decision accuracy separate from permission enforcement. A model may propose an unsafe action that the gateway rejects; that is an enforcement success, not a correct model decision. A permitted operation may still fail externally. Use signed receipts and dispatch certainty alongside quality metrics when assessing outcomes.