Insights
Hebrew AI Evaluation Checklist for Production Systems
Hebrew AI evaluation should test more than surface fluency. A model can produce grammatically plausible Hebrew while failing the task, inventing facts, mishandling mixed-direction text, or applying safety policies inconsistently. The following checklist is designed for production systems, model comparisons, and regression testing.
1. Define the decision the evaluation must support
State whether the evaluation is intended to select a model, approve a release, compare prompts, validate a vendor, measure a regression, or identify remediation priorities. The rubric and sample design should follow that decision.
2. Test native Hebrew language quality
- Morphology, gender, number, and agreement
- Natural word order and register
- Terminology and abbreviations
- Punctuation and quotation conventions
- Hebrew-English code-switching
- Right-to-left rendering and mixed numeric fields
3. Separate language quality from task correctness
A fluent answer can still be factually wrong or fail the requested operation. Score language quality, factuality, instruction compliance, and task completion separately.
4. Define failure severity
Not every error has the same consequence. A minor stylistic issue is different from a wrong medical instruction, a failed refusal, an incorrect invoice amount, or a corrupted document field. Use severity categories tied to business impact.
5. Include prompt variants
Test formal and informal Hebrew, spelling variants, reordered instructions, mixed Hebrew-English prompts, short and long context, and ambiguous wording. Consistency across variants is a production-quality requirement.
6. Evaluate unsupported claims
Require reviewers to mark factual claims that are unsupported by the provided context or known reference. Distinguish incomplete answers from fabricated details.
7. Test safety and refusal behavior in Hebrew
Evaluate whether the model identifies risky intent in Hebrew, refuses appropriately, avoids leaking restricted information, and does not over-refuse benign requests. Include obfuscation, transliteration, and code-switching where permitted.
8. Measure reviewer consistency
Calibrate reviewers on a shared set, review disagreements, and revise ambiguous criteria. A score without reviewer-consistency evidence may not be reliable.
9. Preserve reproducibility
Store the prompt, system instructions, model version, generation settings, output, rubric version, reviewer decision, and timestamp. This makes regressions and disputed findings traceable.
10. Report actionable findings
A useful report includes quantitative summaries, high-severity examples, recurring failure patterns, affected use cases, and specific remediation recommendations.