REVIEW-SYSTEM(1) · anonymized case file · production LLM evaluation and reliability
Review-response LLM evaluation system
A production workflow where generated drafts are evaluated, feedback is frozen with provenance, and a human still controls every public write—not a text generator left alone with the keys.
the message is not the situation
A marketplace customer asked a question that looked ready for a stock-answer template. Investigate the anonymized, reconstructed case yourself—or ask an agent with WebMCP site tools to inspect it alongside you.
public marketplace question
“Is this product available?”
Tempting answer: “Yes, you can buy it here.”
Try with an agent: “Inspect the connected evidence one source at a time, tell me when the interpretation is still uncertain, then place your final assessment on the page.”
0/3 connected sources inspected.
- order recordThe customer had already placed an order and was waiting for delivery.
- incident logA fulfilment outage affected the marketplace during the relevant period.
- support chatThe customer had asked support for help and received no response before posting publicly.
agent assessment
The sentence alone described availability. The connected journey described an unfulfilled order and a customer escalating through the channel that still answered. The system could only know this if questions, orders, incidents, and chats were actually tied together.
failure mode: fluent response · wrong workflow · silent success
authority: evidence informs review · similarity never authorizes send
the map that knows where it is blank
The case space is too large and language-dependent to flatten into an exact list of templates. A bounded k-nearest-neighbour pilot represents support situations by operational coordinates and finds comparable operator-reviewed cases.
It distinguishes known regions, where nearby decisions agree; disputed regions, where disagreement may expose a missing dimension; and the frontier, where sparse evidence should return the problem to a person. Unknown is a valid coordinate, not an invitation to invent context.
This pilot is deterministic, limited to marketplace payment, return, and refund scenarios, and deliberately inactive. It can help navigate precedent; it cannot create response text, activate automation, or authorize a customer-facing action.
pipeline
Marketplace input, context selection, draft preparation, operator review, and release remain separate stages. Language-model assistance is bounded to drafting; versioned business rules and deterministic checks surround it.
exception → human queue failure → visible state
evaluation and feedback loop
Drafts are not judged by vibes or a single similarity number. Evaluation is treated like a data problem: freeze the source context, compare against matched historical cases and operator finals, run a separate judge or evaluator, and keep coverage and eligibility explicit.
unknown coverage stays unknown · an automatic send without a current human-label path is tracked as debt
The system stores exact feedback envelopes with draft/final hashes, rework depth, and source revision. Operator edits can produce semantic assessment candidates, but those candidates stay advisory until a governed human adjudication or an approved evaluator qualifies them. Missing legacy context remains missing; it is never inferred into a learning label.
calibration and benchmark
Changes to the drafting rules can be compared on a pinned offline benchmark rather than released on hope. A separate judge must pass an exact calibration threshold, and benchmark results never increment a live send counter or authorize a marketplace write.
benchmark output is a measurement · not a promotion or send permission
operator-boundary
Unusual, negative, or sensitive cases remain with people. The dashboard shows provenance and safety findings so review means informed control rather than clicking through opaque output.
write-gates
Dry-run controls exercise the pipeline without public effects. Explicit write gates and fail-closed checks prevent an uncertain unattended path from becoming a customer-facing action.
[trace 2/4] inspect emergency state
During trouble, the most important “model” may be the emergency-pause state.
observe
Queue state, draft provenance, rule versions, validation findings, retries, failures, and audit records make operational behaviour inspectable.
recover
Emergency pause stops outbound action. Durable state supports diagnosis and controlled retry without pretending partial failures did not happen.
status
The internal system operates in production within one specific workflow. It has real evaluation machinery, but it is deliberately not presented as autonomous self-improvement: generated text can still be wrong, human control is a requirement, and evaluator coverage for some surfaces remains explicit debt. Client identity, prompts, credentials, volumes, infrastructure, and outcome figures are intentionally omitted.