Grading for translation quality or legibility? A challenge to objective assessment of translation quality in handwritten samples

Research literature on translation quality assessment focuses on substantive criteria for reliable and intersubjective appraisals of translation quality. However, assessments involving handwritten translations (which are still routinely used in many formal testing and certification procedures, competitions, and occasionally in academic translator training) may be also affected by psychological effects related to sample legibility, such as the halo effect and the processing fluency effect. As a result, assessments may be biased by the appearance of the text on the page, a factor which has no relevance to translation quality. This article seeks to establish whether varying levels of legibility introduce bias to assessments of quality by boosting scores (in the case of neat handwriting) or depressing them (in the case of messy handwriting). To test this hypothesis, the results of a nationwide translation competition judged by a panel of professional translators were examined for connections between perceived penmanship quality and perceived translation quality, as measured by actual performance in the competition. The complete set of handwritten translations in the competition (N=38) was assessed for legibility by a panel of independent appraisers. Those legibility scores were then compared to the official results of the competition. The findings reveal a statistically significant difference in legibility scores separating the most successful entries (prize winners) from the least successful entries, suggesting that penmanship standards could be unduly influencing appraisals of translation quality. The finding has relevance not only for the design of appraisal procedures in certification exams or competitions, but also for translation quality assessments performed by instructors in translator training.