Score voice
annotation quality.

#Overview

Built for annotation vendors and ML teams shipping Indic ASR datasets. Gives you a defensible per-clip score, the rule that was broken, and a batch-level pass rate — all in a single JSON call.

What it does

• Compares an annotator's JSON transcript against a gold JSON for the same audio
• Runs ~32 HARD rules (regex / Unicode-script / whitelist) — no judgment, deterministic
• Runs ~14 SOFT rules through an LLM-judge with a 0.85 confidence threshold
• Returns a score in [0, 1], a pass/fail (default threshold 0.97), every violation, and the math behind the score
• Supports single-pair and batch scoring, with per-clip + aggregate stats

What it's not

• Not a transcription service — bring your own annotator + gold pair
• Not language-detection — assumes the framework's Hindi+English (code-switched) target
• Not authoritative — final acceptance is the customer's call; this is a QC signal

#Categories — WER, Punctuation, Tags

Every score response groups violations under three top-level buckets. The portal renders them as three cards; the API returns them under breakdown.categories. Each bucket has its own score, pass flag, and violation list.

Per-category scoring

Each bucket has its own passed flag (threshold 0.97). The top-level score is the original composite — it still considers every violation across all buckets, so the overall pass/fail stays consistent.

Example response (categories field)

copy{
  "score": 0.92,
  "passed": false,
  "breakdown": {
    "categories": {
      "wer": {
        "name": "wer",
        "label": "Lexical accuracy (WER)",
        "score": 0.98,
        "passed": true,
        "hard_violation_count": 0,
        "soft_disagreement_count": 0,
        "rescued_count": 0,
        "violations": [],
        "soft_disagreements": [],
        "rescued": [],
        "details": {
          "blended_lexical_error": 0.02,
          "latin_token_count": 12,
          "indic_token_count": 18
        }
      },
      "punctuation": {
        "name": "punctuation",
        "label": "Punctuation",
        "score": 0.95,
        "passed": false,
        "hard_violation_count": 1,
        "soft_disagreement_count": 0,
        "rescued_count": 0,
        "violations": [
          {"rule_id":"1.6","severity":"hard","message":"Banned character ';'", ...}
        ],
        "soft_disagreements": [],
        "rescued": [],
        "details": {}
      },
      "tags": {
        "name": "tags",
        "label": "Tags & speaker labels",
        "score": 0.975,
        "passed": true,
        "hard_violation_count": 0,
        "soft_disagreement_count": 1,
        "rescued_count": 0,
        "violations": [],
        "soft_disagreements": [
          {"rule_id":"7.2","severity":"soft","message":"Emotion primary label disagreement", ...}
        ],
        "rescued": [],
        "details": {}
      }
    },
    "hard_violations": [...],     // flat list, kept for back-compat
    "soft_disagreements": [...],
    "rescued_disagreements": [...],
    "math": { ... }
  }
}

#Quickstart

Sixty-second copy-paste. Replace the base URL with your deployment.

1. Score one pair

copycurl -X POST https://your-deployment.vercel.app/api/score \
  -H "Content-Type: application/json" \
  -d '{
    "annotator": [
      {"speaker":"Speaker_1","start":"00:00:00.000","end":"00:00:02.500",
       "text":"मैंने कल Delhi से नया laptop खरीदा। [None]"}
    ],
    "gold": [
      {"speaker":"Speaker_1","start":"00:00:00.000","end":"00:00:02.500",
       "text":"मैंने कल Delhi से नया laptop खरीदा। [None]"}
    ]
  }'

Returns:

copy{
  "file_id": "annotator",
  "score": 1.0,
  "passed": true,
  "breakdown": {
    "hard_violations": [],
    "soft_disagreements": [],
    "rescued_disagreements": [],
    "lexical": { "blended": 0.0, ... },
    "math": {
      "lex_component": 1.0,
      "hard_penalty": 0.0,
      "soft_penalty": 0.0,
      "final_score": 1.0,
      "pass_threshold": 0.97,
      "formula": "score = max(0, min(1, (1 − blended_lexical_error) − 0.01·n_hard − 0.005·n_soft))"
    }
  }
}

2. Score a whole batch

copycurl -X POST https://your-deployment.vercel.app/api/batch \
  -H "Content-Type: application/json" \
  -d '{
    "pairs": [
      {"file_id":"clip_001","annotator":[...],"gold":[...]},
      {"file_id":"clip_002","annotator":[...],"gold":[...]},
      {"file_id":"clip_003","annotator":[...],"gold":[...]}
    ]
  }'

Returns per-clip results and a summary:

copy{
  "results": [
    { "file_id": "clip_001", "ok": true,  "result": { "score": 0.987, "passed": true,  ... } },
    { "file_id": "clip_002", "ok": true,  "result": { "score": 0.943, "passed": false, ... } },
    { "file_id": "clip_003", "ok": false, "error": "Failed to parse submissions",
                              "detail": "Top-level JSON must be a list of utterance objects." }
  ],
  "summary": {
    "n_total": 3, "n_scored": 2, "n_failed_to_parse": 1,
    "n_passed": 1, "n_failed": 1,
    "pass_rate": 0.5,
    "average_score": 0.965,
    "avg_hard_violations": 1.0,
    "avg_soft_disagreements": 0.5,
    "pass_threshold": 0.97,
    "elapsed_ms": 412
  }
}

#Auth & limits

#POST /api/score

Score one annotator submission against one gold reference.

copyPOST /api/score
{
  "annotator": [{"speaker":"Speaker_1","text":"मैंने hello। [None]"}],
  "gold":      [{"speaker":"Speaker_1","text":"मैंने hello। [None]"}]
}

#POST /api/batch

Score many pairs in one request. Bad pairs don't kill the batch.

#GET /api/docs

Machine-readable OpenAPI 3.1 spec. Use it to generate clients or import into Postman.

copycurl https://your-deployment.vercel.app/api/docs > openapi.json
# Import openapi.json into Postman / Insomnia / openapi-generator

#Schemas

Utterance

One row of an annotator's or gold's transcription.

ScoreResult

The per-file score envelope.

CategoryScore

One of wer, punctuation, or tags.

Violation

#Scoring math

No black box. The exact arithmetic for every result is returned in breakdown.math.

Why these weights?

• Lexical dominates because the buyer SLA is "WER < 3%". A clean transcription gets close to 1.0 even with a few minor rule slips.
• HARD violations get 2× the SOFT penalty because they're unambiguous — no judgment, no rescue.
• 0.97 threshold mirrors the 3% WER ceiling, so passing the threshold also means meeting buyer's lexical bar.

#Rules: HARD vs SOFT

~70% of rules are deterministic (regex / Unicode-script / whitelist). ~30% need genuine judgment and are routed to an LLM-judge.

Full classification table: docs/rule-classification.md.

#Error codes

In batch mode, parse errors do NOT return 422 — they appear as ok:false entries inside results[] with HTTP 200.