Scoring with guardrails

An AI-assisted admissions scoring workflow with a human at every decision, co-built with the organization's technology lead.

“The question was never whether AI could score it. It was whether we could prove the score was fair.”

Challenge

Admissions had no standardized, scored instrument for written responses on durable skills. Reviews were inconsistent and hard to audit.

Admissions reviewers, and the technology lead building the scoring tool.

A six-skill rubric with anchor tables for scores 1 to 4 and calibration examples per level
Scoring rules: a floor rule, no averaging, score down when between anchors, hold every score of 1 for human review
A prompt-injection guard: applicant text is treated as data, never as instruction
A synthetic blind validation set, one response per skill per level, weighted toward borderline cases, with a separate answer key and agreement metrics (exact, adjacent, directional bias, floor accuracy)

De-identified IDs only. The model never sees a name.
Humans make every admission decision. AI scores are confirmed by staff in the system of record.
Accuracy is benchmarked against human-to-human agreement, not a raw pass mark, because human raters disagree too.
Validation uses synthetic responses, never real applicant data. Slower to build, impossible to leak.

A consistent, auditable scoring process by design. Validation results reported internally in general terms.

Re-run validation each cycle and publish the agreement metrics to the admissions team.