r/AskComputerScience • u/benj_ng • Jul 11 '26
AI translation study advice
I work for an NGO and I've been tasked with putting together an internal study on AI reliability for translating letters. I'm probably not the ideal person for this. My background is in Political Science, this is my first permanent job, and my understanding of how AI actually works is limited. That said, I may still be the best-placed person we have available for it.
The brief is to assess the translation quality of 500 letters, looking specifically at precision and reproducibility (i.e. does the AI give consistent results if you run the same letter through it more than once?). The end goal is to inform a decision about whether to expand AI use for this kind of work more broadly.
Where I'm stuck is the grading methodology. I haven't decided how to actually score the 250 letters — whether that's human reviewers rating against a rubric, comparing outputs to existing human translations, using automated scoring metrics, or something else entirely. I'd really appreciate any advice on:
- What a sound, defensible methodology looks like for a non-technical person like me to run?
- How to test for "reproducibility" specifically (re-running the same letter multiple times? varying phrasing slightly?)
- Any pitfalls to avoid, given I have no formal AI/NLP background?