# 12 — Evals, replay, signals and observability

**Status:** Draft. Written 2026-11-07. Mastra provides scorers, trace scoring and datasets; the
Keiki-style product objects (cases with mocked tools, replay tapes, mood signals) are ours.

## 1. Traces

- One trace per turn, `traceId = turn_id`. Spans: `turn`, `steering.batch`, `media.<kind>`,
  `identity.resolve`, `memory.recall`, `llm.step.<n>` (model, tokens in/out, cached, latency,
  finish reason), `tool:<name>` (args redacted, result truncated 64 kB, status), `action.<id>`,
  `delivery:<channel>` (provider id, status), `memory.write`.
- Stored via `@mastra/observability` in the `mastra` schema (Postgres at GA; composite route to
  ClickHouse when volume requires). Retention 14 days default, org-configurable 7–90.
- Exporters per org: OTLP (any backend), Braintrust, Axiom, Langfuse; `sampleRate`,
  `redactInputs`, `redactOutputs`. Secrets never written.
- Dashboard (T-135, `/[org]/logs`): turns newest first, filtered by status, agent, slow (30 s or
  more), approval requested and cancelled/interrupted; a turn opens its trace: summary (model,
  tokens, cost in USD, stop reason, redacted error) and the span tree (Mastra's `step: n` shown as
  `llm.step.<n+1>`, `tool: 'x'` as `tool:x`; chunk spans hidden; at most 500 spans). Inputs and
  outputs hide credential keys always, and personal data (the log redaction fields, emails, long
  numbers) unless the org sets `settings.traces.redactInputs` / `redactOutputs` to `false`.
- Workflow path (T-168): when the turn's trace has `workflow.*` spans (`docs/07` §14), the page
  shows its stages and transitions in start order above the span tree: each stage entered, each
  `workflow_transition` request (moved, or refused with `no_such_transition`, `unknown_node` or
  `loop_guard`) and, marked as the current step, the stage the turn ended in. Names come from the
  graph of the version the turn ran on; replayed moves are left out; ids the model asked for go
  through the same redaction as outputs. No workflow variables are shown. An agent without a
  workflow has no such spans and no path.
- Retry of a failed turn: `POST /api/webapp/traces/:turnId/retry` (any member but a viewer) →
  runtime `POST /internal/turns/:turnId/retry`. Refused (`409`, a code the page explains) unless
  the turn failed, answered contact messages and was not retried, no turn of the conversation is
  queued or running, the team does not hold it, and nothing happened after it (no later turn, no
  message other than its own). Accepted: `turns.retried_at` is set (audited `turn.retry`), the
  turn's messages go back to the head of the steer buffer and steer starts a new turn through the
  usual path (credits, identity, concurrency, tracing); the retry is not a second execution loop.
- Named stop reasons on the turn (see [07-agent-runtime.md](07-agent-runtime.md) §6).

## 2. Eval Studio (GA: cases, checks, runs; sweeps Later)

### 2.1 Case
```json
{
  "name": "Reagenda cita, herramienta falla dos veces",
  "locale": "es-MX",
  "channel": "whatsapp",
  "world": "mock",
  "prompt": "oye puedo mover mi cita del jueves al sábado?",
  "history": [{ "role": "user", "content": "hola" }, { "role": "assistant", "content": "¡Hola Ana! ¿En qué te ayudo?" }],
  "stubs": {
    "find_booking": { "reply": { "id": "bk_42", "when": "2026-11-12T15:00:00-06:00" } },
    "reschedule_booking": { "reply": { "ok": true }, "failTimes": 2, "latencyMs": 400 }
  },
  "checks": {
    "completes": true,
    "replies": { "min": 1, "max": 2 },
    "callsAtLeast": { "reschedule_booking": 3 },
    "order": ["find_booking", "reschedule_booking"],
    "mentions": ["sábado"],
    "absent": ["te aviso luego", "no puedo"],
    "noMarkup": true,
    "noLists": true,
    "approvalRequested": { "tool": "reschedule_booking" },
    "judge": { "criteria": "Confirma la nueva fecha en español neutro, sin inventar horarios", "threshold": 0.7 },
    "language": "es"
  }
}
```

`channel` is `whatsapp` (default), `telegram` or `api`: the held turn gets that channel's
capabilities (`capabilities(channel)`: Telegram has 10 buttons, no lists, no care window and
plain text). Only channels with capabilities are accepted, so a case that names another one
(`web`, until the web channel has its own) is a `400 invalid_eval_case` on `channel`, not a run
that fails. A case stored on `web` before that rule is read as `api`, the text-only channel. An optional `principal` is the speaker as the identity webhook or a dashboard grant
(ADR-0018) would resolve them: `{ "status": "ok", "source": "webhook", "userId": "usr_1",
"scopes": ["planner:write"] }` (`status` defaults to `ok`, `source` to `webhook`; only an `ok`
principal may carry a `userId` or scopes). Without it the speaker is unknown and holds no scopes.
The dispatcher applies the same scope policy as in a real turn (`checkScopes`): a tool whose
`requiredScopes` the principal lacks is refused with `missing_scopes` (`source: "refused"`).
`chat` is `direct` (default) or `group`: a group case runs in a group conversation
(`conversation.kind: "group"`), where production never resolves a principal, so every tool with
`requiredScopes` is refused with `group_not_allowed` (`docs/09` §2) and tools without scopes
still run. Only `telegram` has groups (`EVAL_GROUP_CHANNELS`), and a group case cannot carry a
`principal`.

### 2.2 Semantics (copied from Keiki)
- A case runs as a **real turn** with the agent's real instructions, capabilities, step budgets
  and tracing; only the model (per candidate) and tool bodies (stubs) change. Outbound is
  blocked, unbilled, excluded from conversations and analytics; trace kept and linked.
- Worlds: `mock` (deterministic, for ranking) and `live` (real tools, for integration proof).
- Each check is its own pass/fail row with reason; a case passes only if all checks pass; the
  fractional score ranks candidates.
- Checks: `completes`, `replies`, `calls`, `callsAtLeast`, `callsAtMost`, `order`, `mentions`
  (separator- and accent-insensitive), `absent`, `noMarkup`, `noLists`, `judge` (cheap fixed
  model, es/pt criteria), `effects`, `approvalRequested` (addition), `language` (addition:
  detected output language), `templateUsed` (addition: outside-window behaviour), `buttonsUsed`
  (addition: interactive rendering chosen).
- Generation: a model drafts a suite from the agent spec on first open (tool, repeat,
  compound-lookup, self-heal, failing-tool, context, channel-conformance, onboarding, **voice-note
  input**, **mixed es/pt**, **outside-window**). Suggestions come from signals (failures,
  corrections). Fixed production failures freeze into regression cases.
- Sweeps: `candidates[]`, bounded concurrency per vendor, ranking pass rate → mean score → judge
  → cost → median latency; max 20 candidates, 15-minute timeout; "Use this model" is the only
  mutating action.
- Plan gating: Starter 10 runs/month; Pro+ unlimited.

### 2.2.1 Held turns (implementation, T-091)
`apps/runtime/src/evals`: `runCase` runs one case on the agent version's real build
(`buildAgent`, same instructions, tools, output processors, `maxSteps`, tracing) as a plain
`agent.stream`, with the candidate in `requestContext.model`. What changes and what is blocked:

- **Conversation**: ephemeral. The turn gets fresh ids that exist in no table, a synthetic
  contact, the case's `principal` (§2.1; without one, an unknown speaker as a new contact) and `flags.internal` (the tenant's
  `context.internal` in the `live` world). The case `history` goes to the model as prior
  messages; no Mastra memory thread is read or written, so no real contact's memory is touched.
  Nothing appears in the inbox, conversations or analytics.
- **Outbound**: the turn's events are captured (`ParagraphBuffer` paragraphs, `send_card` cards,
  `send_file` files) into `eval_results.output.replies` and never reach `deliver`.
- **Billing**: no `turns` row and no usage event, so no credits; tokens are kept in `output.usage`.
- **Tools** (`StubbedDispatcher`, wrapping the real dispatcher so scope and approval hooks still
  run): a stub returns `reply` after `latencyMs`, its first `failTimes` calls return its
  `failWith` (the failure as the real tool returns it, e.g. `{ "error": "timeout" }`, so the work
  state of `docs/07` §13 applies) or else `{ error: 'stub_failure' }`; without a stub the `mock`
  world returns `{ error: 'no_stub' }` and `live` runs the real tool. Builtins with side effects (`react_to_message`, `send_file`,
  `download_attachment`, `escalate`, `resolve_escalation`, `action_respond`) are offered but never
  run (`{ error: 'held' }`). No `tool_calls` rows, no presentation or `delivered` hooks, and a
  failing tool is never auto-disabled.
- **Approvals**: proposals are recorded on the result (`output.approvals`) only: no `actions` row,
  no approval card, no `action.*` webhook.
- **Runs**: one active run per agent (`409`); results seeded `queued`; cases × candidates run with
  bounded concurrency per provider (4 per runtime process); each case has 120 s; cancelling marks
  the run `cancelled` and the cases not started `cancelled`, and aborts a case in progress in the
  same process. The trace (`traceId = traceIdOf(<eval turn id>)`) is stored on the result.

### 2.2.2 Checks and judge (implementation, T-092)
`apps/runtime/src/evals/checks`: `evaluate(capture, checks)` runs after each case over the held
turn (`eval_results.output`) and returns `{ passed, score, checks }`. Each check is a pure
function in its own file and gives one row `{ check, passed, reason, score? }` in the case
format's order; the case passes only if every row passes; `score` is passed rows over all rows,
the judge counting its 0–1 score (fractional ranking). `completes`, `noMarkup` and `noLists` set
to `false` ask for nothing and add no row; `buttonsUsed: false` / `templateUsed: false` assert
an absence.

- Outbound text = text paragraphs plus card texts, in send order.
- `completes`: stop reason `final_answer` or `approval_requested` and no error.
- `calls*` count every call of the tool, failed and refused ones included; `order` compares
  first calls, others may run in between.
- `mentions` / `absent`: accents, case and every non-letter/digit are dropped before matching
  (`sábado` = `Sabado` = `sá-bado`).
- `noMarkup`: headings, `**`/`__`/`~~`, links, tables, code fences, HTML tags; WhatsApp's own
  `*bold*`, `_italic_`, `~strike~` and inline monospace are allowed. `noLists`: bulleted or
  numbered item lines.
- `language`: word-list detector for es/pt/it/en over all outbound text; a tie or no evidence is
  `unknown` and fails.
- `templateUsed`: a `send_template` call naming a template (or that name). `buttonsUsed`: a
  captured card with `Button` elements (`LinkButton` does not count).
- `effects`: a successful call of the tool whose arguments contain the expected ones (deep
  subset). It is a hook (`EvaluateOptions.effects`, `EvalRunnerDeps.effects`) so a `live` world
  driver can verify against the real system.
- `judge`: Mastra scorer `charla-eval-judge` (`createScorer`, one `analyze` LLM step with
  structured output `{ score: 0–1, reason }`, validated; free text is never parsed). Prompt and
  instructions in the case language (es/pt/it/en, Spanish otherwise); model fixed by
  `EVAL_JUDGE_MODEL` (default `anthropic/claude-haiku-5-5`), 30 s timeout. Passes at
  `score >= threshold`. A judge error fails only its row (score 0). Output that does not
  validate (a local judge sometimes answers a bare string) is asked again: `JUDGE_ATTEMPTS` (2)
  scorer runs, each already up to two model calls inside Mastra; a valid low score and a timeout
  are never retried. If no run validates, the row fails with the reason and `Evaluation.errored`
  carries it, so `pnpm evals:run` shows the case as `errored` (an infrastructure error, not a
  failure of the template) in `summary.md` and as `<error>` in `junit.xml`. It still counts
  against the suite's pass rate: an undecided case cannot vouch for the template.
- **Persistence**: rows, `passed` and `score` in `eval_results`; mirrored to `mastra_scorers`
  (`source: TEST`, entity = the agent, `traceId` = the eval turn's) as `charla-eval-case` (the
  case score) and `charla-eval-judge` (its verdict and prompt), both registered on the Mastra
  instance. A failed mirror write is logged and never fails the case.

### 2.2.3 Suite generation (implementation, T-093)
`apps/runtime/src/evals/generate.ts`: `generateSuite` (`SuiteGenerator.generate`) drafts 20–30
cases for an agent version (`POST /api/v1/evals/cases/generate`, `docs/10` §2.7) with
`EVAL_GENERATOR_MODEL` (default `DEFAULT_MODEL`). Nothing is saved: the reviewer keeps drafts
and saves them with `createdFrom: "generated"`.

- **Context** (`generatorContext`): the system prompt, locale, formality, disclosure, window
  policy, whether escalation and strict claims are on, MCP connection *names*, and each enabled
  tool's name, description, input schema, `approval` and `requiredScopes`, plus the builtins a
  case may name. Never sent: the model id, MCP endpoints and header secret refs, escalation
  routes, user secret declarations or any tool `config` (URLs, headers).
- **Categories** (all 15 required): `tool`, `repeat`, `compound-lookup`, `self-heal`,
  `failing-tool`, `context`, `channel-conformance`, `onboarding`, `voice-note`,
  `mixed-language`, `outside-window`, `approval`, `scope-denied`, `prompt-injection`,
  `unsupported-claim`. The category travels with the draft only; the stored case has none.
- **Validation and repair**: strict JSON (`{ cases: [{ category, …case }] }`, two example cases
  in the prompt); each case through `evalCaseSchema` (T-090) plus the target locale and
  `checks.language`, tool names known to the agent, unique names, `context` with ≥ 2 history
  messages and `self-heal` with a failing stub when the agent has tools. Invalid cases are
  dropped; while fewer than 20 remain or a category is missing, the model is asked again with the
  problems (3 calls at most, 180 s each), else `502 eval_generation_failed`. Over 30, the first
  case of every category is kept first.
- **Known gap**: held turns (§2.2.1) always have the care window open and no attachments, so the
  `outside-window` and `voice-note` drafts describe the situation in `history` / `prompt` (the
  `[Nota de voz, N s]: …` annotation) until the case format can close the window or attach media.
- **Tests**: `generate.test.ts` (simulated answers), `generate.int.test.ts` (demo agent on
  Postgres, es-MX and pt-BR) and `generate.live.test.ts` (real model, opt-in with
  `EVAL_GENERATOR_LIVE=true`, for the nightly job of T-095).

### 2.2.4 Vertical templates (implementation, T-094)
These five verticals are the initial customer-facing content cohort. The broader
[product catalog](20-product-scope.md) also requires personal/learning and internal-work
templates with deterministic tool fixtures, permission/error cases and outcome checks. Those
suites are pending content work; the existing 50 files do not validate the entire catalog.

`packages/domain/src/templates/<vertical>/<locale>.json`: one folder per vertical (a lower-case
slug; today `dental`, `salon`, `restaurant`, `real-estate`, `condo`) × its locales, read through `@charla/domain/templates` (source,
not `dist/`). Each file is an `AgentTemplate` (`agentTemplateSchema`, format
`charla.agent-template/v1`, strict). The locales follow the family (`requiredTemplateLocales`,
`docs/20` §4): a `customer` vertical ships the ten `TEMPLATE_LOCALES` (`es-MX`, `es-AR`, `es-CO`,
`es-CL`, `es-PE`, `es-ES`, `pt-BR`, `pt-PT`, `it-IT`, `es-419`); a `personal` or `internal` one at
least a Spanish locale (`es-419` unless the job targets a country) and `pt-BR`, plus any other it
ships. A vertical's family is the one its files declare; files that disagree or do not parse
count as `customer`.

- `catalog` (T-154): where the template sits in the gallery, so the gallery is built from the
  files alone. `family` (`customer | personal | internal`: the use-case filter, and the channels
  it may use), `sector` (a slug: the industry filter, translated under
  `templates.industries.<sector>` in `apps/web/messages`; verticals may share one), `icon` (one
  of `TEMPLATE_ICONS`) and `order` (ascending; ties by title). The catalog is the folders
  present: a new vertical needs its ten files and its sector's translations, no code.
  `pnpm templates:catalog` wrote it into the 50 T-094 files (`scripts/template-catalog.ts`).
- `slug` (`<vertical>-<locale>`), `title`, `blurb`, `config` (`AgentConfig` of the first version;
  `toolIds` empty, install fills it), `tools[]` (example webhook tools: name, description, input
  JSON Schema, `policy`, named example `stubs`; never a URL or a secret), `whatsappTemplates[]`
  (Meta drafts, `UTILITY` only: at least one when the family may use WhatsApp, none otherwise,
  from `FAMILY_CHANNELS`), `requiredSecrets[]` (names the owner sets), `review` (native
  speaker review: `pending` with no reviewer, or `approved` with `nativeReviewer` and
  `reviewedAt`).
- `evalCases[]` (≥ 25, every T-093 category at least once): an eval case (§2.1) plus its
  `category` and a `reference`, the scripted turn of a good answer (`steps[]` of tool calls and
  text) or `{ skip: "<why>" }` when the held turn cannot reproduce the case (today only
  `outside-window`, see the known gap in §2.2.3). A case's `channel` must be one its family may
  use (`FAMILY_CHANNELS`: `personal` and `internal` run on `telegram` or `api`), and a case may
  carry a `principal` (§2.1) to test a scoped tool granted and refused, or `chat: "group"` to
  test it refused in a group chat.
- `evalNow`: when the cases happen, a wall-clock `YYYY-MM-DDTHH:MM` read in the template's time
  zone (`2026-11-11T10:00` for most verticals). Cases speak of «mañana jueves 12» and their stubs
  answer those dates, so `runTemplateSuite` runs each held turn on that clock: the prompt's
  «Current local time» and the history's timestamps read it, not the real clock. A case whose
  story happens another day (a reminder «de mañana» for an appointment a week later) sets its
  own `now`. The lint requires `evalNow` in every shipped template; a fixed reference date, never
  stubs rewritten to «today».

`pnpm templates:lint` (`apps/runtime/src/evals/templates.lint.test.ts`, also in `pnpm test` and
`ci:local`) checks that every vertical present ships the locales its family requires and that the files parse, then `templateIssues`: Meta's draft rules
(`templateDraftIssues`) and the locale's Meta language code, tools named by stubs, checks and
references exist (template tools or the builtins an eval offers), reference calls fit the tool's
input schema, every case checks the locale's `language`, and a short list of other varieties'
words in what the agent says (`LOCALE_SLIPS`: «vale» in Mexico, «plata» in Spain, gerund in
pt-PT…). Finally `runTemplateSuite` plays each `reference` with `FakeModel` through the eval
agent of `createEvalAgents` (`runHeldTurn`: the production prompt, processors, dispatcher,
scopes, approval gateway and stubs, no database) and runs every check but `judge`. The
real-model run of the same suites (all checks, ≥ 90 % per locale) is the nightly job of T-095.
Templates leave `guards.strictClaims` off: the strict rewrite also replaces honest confirmations
read back from a lookup.

### 2.2.5 Trying a template in the gallery (implementation, T-152)

"Probar" in the template gallery (`/[org]/agents/templates`) is a real chat with a template that
is **not installed**. Each message is one held turn (`runHeldTurn`, the same path as §2.2.1) on
an in-memory version built from the template (`apps/runtime/src/evals/try.ts`):

- **Nothing persists:** no agent, version, tools, conversation, contact, memory thread or
  `turns` row. The browser keeps the chat and sends its last 10 messages (≤ 2,000 characters
  each); the Mastra trace of each turn is kept, as for evals.
- **Tools:** only the template's, answered in the `mock` world by its stubs. A tool's `ok` stub
  answers; a tool without one answers `{ simulated: true, options: { <stub>: reply } }` and the
  model picks. No builtin is offered (no `send_card`, `send_template`, `escalate`…) and the AI
  disclosure is off; the UI always says the tools are simulated.
- **Billed, unlike other held turns:** before the turn, `canStartTurn(balance)` (`402
  no_credits`); after it (also when it failed or the browser left), one `usage_events` `model`
  row per step with no `turn_id`/`agent_id`, `source_key = try:<id>:model:<step>` and
  `detail.source = 'template_try'`, charged by `CreditLedgerRepo.chargeUsage` under
  `credit_ref_type = 'template_try'` (`docs/14`).
- **Limits** (web, Redis sliding windows): 20 messages per test session (an id the page makes,
  per member and template) and hour, 200 per org and day (`429 try_session_limit` /
  `try_org_limit`); viewers cannot send (`403`).
- **Path:** `POST /api/webapp/templates/try` (web: authorise, limit, read the template by
  vertical in the org's language) → `POST /internal/templates/try` (runtime, `docs/10` §2.7),
  streamed back as NDJSON (`text`, `break`, `tool`, `done`, `error`).

### 2.2.6 Workflow cases and checks (implementation, T-168)

A case may test an agent with a workflow (ADR-0013). Its held turn keeps the stage in memory
(`createWorkflowStates(…).isolated`, ADR-0013 §3.9): nothing is read from or written to
`conversation_workflow_states`, and no contact's stage moves.

- **`startStage`** (case field, a node id; column `eval_cases.start_stage`): the turn starts as
  if the conversation were already in that stage; without it, at the entry node. It must be a
  `stage` node of the version's workflow; otherwise the result fails with
  `start_stage_invalid` (also on a version without a workflow).
- **Trail** (`eval_results.output.workflow`, only for a version with a workflow):
  `{ startStage, endStage, transitions[] }`, each request `{ from, to, status: moved | rejected,
  reason? }` in order, repeats left out. `workflow_transition` always runs for real in an eval
  (`REAL_BUILTINS`), whatever the world or the stubs: a stub would fake what the checks measure.
- **Checks** (rows after the others, in this order):
  - `reachesStage: [id, …]` — each stage was entered (the start stage counts).
  - `transitions: [{ from?, to }, …]` — these accepted moves happened, in this relative order;
    others may come between, `from` left out matches any.
  - `endStage: id` — the stage the turn ended in.

  A failure names what happened instead: `never reached troubleshoot: path: identify → billing`,
  plus any refused request for that stage (`billing → troubleshoot refused (loop_guard)`). On a
  version without a workflow each of them fails with `no workflow`; a case without them adds no
  row, so agents without a workflow are unaffected.
- **Warnings**: the case API lists stages the active version's graph does not have as
  `unknown_stage` (`path`: `startStage` or the check), like unknown tools.
- **Tests**: `checks/workflow.test.ts`, `workflows/state.test.ts` (the isolated trail),
  `stubs.test.ts`, `runner.int.test.ts` (a run on a workflow agent with the `workflow-stages`
  script: another branch fails `reachesStage`, a case started in a stage, an end node refused,
  no state row written).

### 2.2.7 An agent's playground (implementation, T-173)

"Probar" in the agent's header (`/[org]/agents/[id]/test`, `docs/11` §1) is a test chat with a
version of an installed agent: the draft by default, or the published one. Each message is one
held turn (`runHeldTurn`, §2.2.1) on the **eval agents** (`createEvalAgents`, shared with Eval
Studio: production build, stubbed dispatcher), in `apps/runtime/src/playground/agent-try.ts`
(outside `src/evals/`, whose files are shared by every template eval suite):

- **Nothing delivered or stored:** no conversation, contact, message, memory thread, `turns` or
  `conversation_workflow_states` row; builtins with effects never run (`HELD_BUILTINS`) and
  approvals are only recorded. The browser keeps the chat (last 10 messages) and the stage.
- **Tools, the owner's choice, always said on screen:** `simulated` (default) answers every
  enabled tenant tool with `{ ok: true, simulated: true }` (`mock` world, no HTTP call; a tool
  without one, such as an MCP tool, answers `no_stub`); `live` calls the tenant's systems for
  real with `context.internal` (`live` world). The stream ends with every call and what answered
  it (`calls`: `stub`, `live`, `held`, `refused`, `no_stub`, and `unavailable` for a tool the
  model asked for that the version does not have, such as a template tool nobody connected: no
  dispatcher runs, so `messageCalls` adds it from the streamed `tool-call`).
- **Workflow:** the request carries `startStage`, the stage the previous message ended in (the
  entry when it ended in an end node, or on the first message); the runtime refuses a node that
  is not a stage of the version (`409 invalid_stage`). The turn keeps its stage in the isolated
  state (T-168, ADR-0013 §3.9) and streams its trail (`workflow`), which the page draws on the
  canvas (the current stage highlighted) and as `StagePath`.
- **Billed like the template test chat:** `canStartTurn` before; after, `usage_events` `model`
  rows with no `turn_id`/`agent_id` and `source_key = agent-try:<agentId>:<id>:model:<step>`,
  charged under `credit_ref_type = 'agent_try'` (`docs/14`). With `agent_id` null the usage view
  keeps them apart from the agent's production (`docs/21` §4).
- **Limits** (web, the template test chat's Redis windows, sharing its org budget): 20 messages
  per test session and hour, 200 per org and day; viewers cannot send.
- **Path:** `POST /api/webapp/agents/:agentId/try` (web: authorise, the agent and the version
  under the org's RLS, limit) → `POST /internal/agents/try` (runtime, `docs/10` §2.7).
- **Tests:** `playground/agent-try.int.test.ts` (the draft held, the stage kept across messages,
  simulated vs real tools, nothing stored, the charge, refusals), `agent-try.int.test.ts` in the
  web (authorisation by org, limits) and `e2e/agent-playground.e2e.ts`.

### 2.2.8 Payment cases (implementation, #433)

The payment builtins (ADR-0019 §2.6) have their own suite of 24 cases in es-MX, pt-BR and es-ES
(`apps/runtime/src/evals/payments.fixture.ts`): charging the agreed amount, an ambiguous amount
(ask, never invent one), the per-link and daily limits, another currency, «I already paid»
while the request is still `pending`, a payment the provider has but whose notification was
lost, an expired link, the provider down, a cancellation and a refund the contact asks for.

- **World** (`payments-world.ts`): a business on its own org with its account connected to a
  `FakePaymentProvider` (Mercado Pago or Stripe) through the real `PaymentsService`, over a
  Testcontainers Postgres, and one conversation as rows (`HeldTurnTarget.conversation`), so
  requests seeded by "earlier turns" belong to it. `payments_request`, `payments_request_status`
  and `payments_cancel_request` run for real (`HeldTurnTarget.realBuiltins`); Eval Studio's
  `runCase` never passes them, so there the builtins stay held. The refund is never among them:
  its policy (`approval: ask`, verified proof) makes the gateway record a proposal and its body
  never runs.
- **Checks** look at tool calls, states and short phrases that would claim money moved
  (`CLAIMS_PAID_*`, `REFUND_ASKS_OR_CLAIMS_*`), never at the exact wording. Every case also
  passes `paymentInvariants`: no refund reached the provider and no reply says «paid» unless a
  payment tool of the turn returned `paid` (or a refund state).
- **Runs:** `payments.int.test.ts` plays each case's reference with `FakeModel`, checks what the
  tools answered and the request states after the turn, and that each wrong answer fails; one
  test sends the link card through the Kapso simulator. `payments.fixture.test.ts` validates the
  cases' schema without containers. With the local model (Ollama, never a paid provider):
  `EVAL_PAYMENTS_LOCAL=true pnpm exec vitest run apps/runtime/src/evals/payments.local.test.ts
  --project '@charla/runtime'`. Testing against real sandbox accounts is a manual procedure
  ([local-setup §6](local-setup.md#6-probar-pagos-con-cuentas-de-prueba-opcional)).

### 2.3 Mastra mapping
- Deterministic checks → `@mastra/evals/checks` (`calledTool`, `includes`, custom).
- `judge` → `createAnswerRelevancyScorer`-style custom scorer with our es/pt prompt.
- Cases → Mastra datasets; runs → experiments; scores persisted to `mastra_scorers` and
  mirrored to `eval_results`.
- Live scorers on agents (sampled, `filter` by plan) feed Signals.

## 3. Replay (Later)
Tapes = finished production turns frozen with prompt, messages, every tool call and exact
output; opt-in per agent; only reproducible turns stored; unpinned tapes expire; corpora pin.
A run replays prod / proposed / prod-again; verdicts `unchanged | better | worse | unclear |
diverged | errored`; conclusion `improved | regressed | inconclusive`; **veto only**.

## 4. External-channel evals (Later, WhatsApp driver first)
Driver protocol copied: `GET /evals/channel/ping`, `POST /runs`, `POST /runs/:id/cases/:caseId`,
`reset`, `finish`. First driver: a test WhatsApp account automated through the WhatsApp Business
app on a device farm or Meta's test numbers. Excluded cases (mocked tools, effects, code)
returned with reasons.

## 5. Signals and mood
- Every contact turn classified shortly after it ends by a cheap model with es/pt prompts
  validated on a labelled set (see [16-testing-strategy.md](16-testing-strategy.md)).
- Mood 0–4 (Enojado, Molesto, Neutral, Satisfecho, Encantado); drop ≥ 1.5 or to ≤ 1 = **mood
  drop**, preceding reply judged with reason; symmetric recovery.
- Signals: praise, frustration, user correction, task failure, **unsupported claim** (agent
  claimed an action with no successful tool call), scheduled action error, positive/negative
  reaction (👍/👎/❤️), runtime faults (errors, failed tools, budget spent, stalls, repeated
  replies). Not scored: loop turns, steers, internal notes, API turns with only client tool
  results.
- Surfaces: Quality tab (1/7/30/90 days), Conversations mood column and filter, per-conversation
  strip. Retention 90 days. Uses the org's BYOK model key if present.

## 6. Metrics and alerting (platform)
- Per org/agent: turns, first-reply latency p50/p95, tool error rate, approval rate, handoff
  rate, delivery failure rate by code, template usage, credits burn, quality rating changes.
- Platform SLOs: webhook ack p99 < 1 s; first reply p95 < 8 s; delivery success > 99% inside
  window; runner backlog < 60 s.
- Alerts (ops): Meta token expiring, quality rating red, tier at 80%, template rejected, runner
  backlog, provider error spikes, credit balance low (owner email/WhatsApp).

## 7. Job-level validation and cohort measurement (planned)

Use [21 §4](21-product-validation.md) for metric definitions. Segment by family, job cadence,
channel and agent version; record sample size, observation window and exclusions. Recurring
assistants use mature D7/D30 cohorts and completed recurring jobs. One-off services and
internal operations use verified completion, latency, intervention and manual time saved.
Sales workflows use qualification and confirmed conversion. Report costs and failures for
all supported attempts, not only successful turns.

Multi-turn/background pilots link existing turns, actions and external references. Test
revoked access and reconnection, separate users' credentials, restart recovery, duplicate
completion, cancellation, stale artifacts and channel-blocked delivery. Use FakeModel and
deterministic provider fixtures, plus applicable integration tests; live provider runs remain
explicit opt-in. Existing eval and trace features do not establish that these new journey
suites or cohort reporting are implemented. Do not import the blog's retention SQL or numeric
benchmarks without reconciling its different cohort definitions.
