Evaluation

Measure what
your workflow needs.

A correct field, a relevant result, and an answer with the right citation are different tests. Our evaluation framework measures task quality alongside latency, processing cost, and human review.

Evaluation dimensions
DimensionWhat to measureWhat to disclose
Task qualityField accuracy, routing errors, ranking relevance, or document structureDataset composition, sample size, held-out split, and baseline
ReliabilitySchema validity, missing outputs, unsupported cases, and retry rateFailure taxonomy and behavior on incomplete inputs
LatencyEnd-to-end p50 and p95 under defined concurrencyHardware, input sizes, warm-up, batch size, and load
CostInference and review cost per accepted resultPricing unit, retry costs, and review assumptions
EvidenceCorrect source spans, page references, and supported answersAnnotation process and human review method
THE PROCESS

Define. Compare. Stress-test.

  1. Choose a narrow task and document the current workflow baseline.
  2. Build an authorized dataset covering typical inputs and difficult cases.
  3. Separate training, development, and held-out evaluation examples.
  4. Compare model approaches at the same quality target and operating conditions.
  5. Publish limitations and run a shadow pilot before production promotion.
Download evaluation template
Data boundaries

Evaluation starts with authorized data and an agreed scope. Data access, retention and processing requirements are defined before a collaboration begins.

Build what comes next.

Talk to Istren