Translation Quality Measurement Approaches

Six approaches to measuring translation quality is a short article published several years ago on the Medical Design Online web site (the article is still available).

The six approaches mentioned are:

1. Society of Automotive Engineers (SAE) J2450
2. Localisation Industry Standard Association (LISA) QA Model
3. ForeignExchange Translations' Multilingual Compliance Process
4. American Translators Association accreditation program
5. Deutsches Institut für Normung DIN 2345
6. Six sigma quality initiative

The article contains some further details about these differing approaches to quality.

Standards and Models

Better Localization has an interesting article (Standards and models, why should we care?) about how the application of various standards and models would be beneficial in the localization industry.

They are not goals, but tools to achieve goals

The standards described briefly in teh article are ISO 9001:2000, CMMI, BSC, PCMM, and their integration.

ISO 9000 Certification, and Other Methods of Certifying the Quality of LSPs

A few days ago Common Sense Advisory published an article (Looking for Language Service Providers in All the Right Places) on how customers may make sure that LSPs (Language Service Providers) meet their business needs.

...training and certification are good ideas that we believe every buyer ultimately will demand of their suppliers.

Among the interesting things mentioned in the article there is the fact that only 10% of LSPs have ISO 9000 certification.

The article ends up promoting Common Sense Advisory's own certification service for LSPs, which itself looks interesting:

We take a financial auditor's approach to the question of certification -- such firms check a company's compliance to Generally Accepted Accounting Principles (GAAP). Similarly, our 24x7 program looks for "a" process that is used consistently and constantly throughout an LSP's organization. That process can be ISO, defined by a client, internally developed, or any combination of these.

Our Article to Appear in Multilingual

The title of the article is "Translation Quality Measurement: Using the Translation Quality Index to assess the quality of translations", and deals with such questions as:

• Why measure translation quality?


• Why are translations so difficult to evaluate? What methods are available to help assess translation quality?


• What is the Translation Quality Index (TQI)?


• Why measure errors?
  

The article is scheduled to appear in the June 2006 issue of Multilingual, which should also contain other articles about translation quality.

UPDATE

(4/26/06)
The article will actually appear in the July/August issue, not the June issue.

CEN Approves Translation Quality Specification

Common Sense Advisory has an interesting article about the approval, by CEN (the European Committee for Standardization), of EN 15038 quality standard for translation service providers.

The article also compares EN 15038 with ASTM 15.48, then goes on and criticizes the antiquated procedures that were followed in drafting the standard, and above all the lack of transparency in these procedures.

How To Set Up a Language Quality Measurement System

The six phases for setting up a robust quality system:

1. Design
   
   • - Collect a corpus of good and bad translations
     
   • - Analyze the corpus to identify critical-to-quality errors (error definition)
     
   • - Decide what to measure (error categorization)
     
   • - Assign a weight to various types of errors (severities)
     
   • - Define an error threshold
     


2. Calibration
   
   • - Pilot,
     
   • - test, and
     
   • - adjust
     until the system works in an objective, repeatable, and reproducible way
     


3. Sampling
   
   • - Sample selection criteria (e.g. random, systematic)
     
   • - Sample size, confidence intervals, margins of error
     
   • - Cost considerations (find the point of diminishing returns)
     


4. Measurement
   
   • - Calculation of a Translation Quality Index (TQI)
     
   • - Evaluation must be
     
     • * repeatable,
       
     • * reproducible,
       
     • * objective
       
   • - Use of independent auditors
     
   • - Provide formal training
     


5. Statistical Analysis
   
   • - Use of control charts
     
   • - Investigate results
     
   • - Rule out special causes
     


6. Process Improvement
   
   • - Take corrective actions
     
   • - Compare the TQI values before and after a process change to check for actual process improvement
   
   

Writing for Translation

An aspect of translation quality that is often not mentioned, is how much the quality of the SL affects the quality of the translation.

Courses on "Writing for Translation" such as this one are often popular, and, when done right, also useful, as technical writers may learn why, for example, concatenated strings are so much trouble for us.

One thing I saw when I worked in a large business software company, however, was that technical writers often resist any attempt to introduce controlled English, to limit their terminological choices, to introduce stringent editing rules, or, more in general, anything that they perceive as limiting their freedom and creativity.

"About Translation" Locked by BlogSpot

I have another blog on BlogSpot, About Translation, where I write more in general about translation and things of interest for professional tanslators.

Unfortunately, since last week my blog has been locked by Blogger, as their algorithms have (wrongly!!!) deemed it a "spam blog" (no idea why: it is clearly a legitimate blog, but until some human person takes a look, it remains locked)... so while the blog is still up, I cannot post anything new.

If the situation persists, I'll move the blog to some other service.

UPDATE: About Translation has finally been checked by a human member of the support team, white listed, and unlocked.

Brave Comment on the Blog

We have just received a comment on our blog, from "Anonymous":

That's rich, a blog about quality in translation operated by a Lionbridge employee. Lionbridge is a fraudulent company that doesn't pay its translators.

To which I have a few things to say:

1. This blog is completely independent from Lionbridge; it was set up to complement our web site TranslationQuality.com with an area where we could discuss issues related to translation in a more dynamic way than on the main web site (where we make available more information about our work on translation quality, such as our presentations and articles).
   
2. Franco is indeed a Lionbridge employee. I am not: I have my own small translation company, and we provide our services to several larger translation companies. Among the services we provide on request is translation quality assessment services, including helping other companies improving their translation quality systems.
   
3. This information is clearly available both on the "Profiles" section of this blog and on our main web site.
   
4. I suggest that if "Anonymous" has a complaint against Lionbirdge, he should contact Lionbridge directly, and, if not satisfied, take the appropriate legal steps. On the other hand, this blog is not an appropriate venue for such complaints.
   
5. While Franco and I are clearly named in our blog, "Anonymous" has not the courage of signing his message. That, to me, says all there is to say about the merit of his complaint.
   

Any further anonymous abusive messages will be deleted from this blog without comment.

Revision, Editing and Proofreading

The February issue of the ATA Chronicle has just arrived, and with it an interesting article on translation quality control: "A Second Pair of Eyes: Revision, Editing and Proofreading", by S. Alexandra Russell-Bitting.

After introducing herself and her experience, the author devotes sections of her article to Quality Control in Translation, Revision, Editing, Proofreading, and Managing Quality.

Apart from other things, the article could be very useful in that it differentiates between activities that are often lumped together, confused with each other, or misnamed.

In this case, "Revision" is defined as "checking a translation for accuracy and style", going through the text sentence by sentence and comparing SL and TL.

"Editing" is described as "more creative work than revision [where] you have the freedom to make improvements in the text for readability"

"Proofreading", finally, according to Russell-Bitting is "a final check before publication".

When I am asked by a translation company to "proof" or "edit" a translation, I always have to ask what, exactly, they want me to do: what's proofreading to someone is called editing by a others, and revision is confused with both things. I hope this article may help differentiate between these important translation quality control activities.

A Simple QC Technique for Bilingual Files

One of the most insidious kind of errors happens when a negation is translated in the affirmative, or vice-versa. Insidious because sometimes such errors are difficult to spot, as the sentences that contain them, on their own, probably still make sense.

However, when translating using an environment, such as TM program, which produces bilingual files, there is a simple quality control technique that, while not 100% effective, still helps avoiding this type of errors. The following example works for the English-Italian language pair, but it could easily be adapted to other languages:

1. Once the translation (or editing) is complete, search for word "not" throughout the bilingual file
2. For each occurrence of the word found, check the corresponding Italian segment, to make sure that the negative meaning is preserved
   
3. Once completed the search for the English "not", repeate the search for the Italian "non", so as to make sure that a negative meaning has not cropped where the English instead has an affermative sentence
   

Of course this is not 100% foolproof, but it can be improved by adding other relevant words (such as "cannot", "failed", "impossible", etc.) to the search.

Why Translation Quality Measurement?

You cannot manage what you cannot (or do not) measure; yet, this is precisely what most translation quality control processes try to do.

A Method for Predicting the Productivity of Machine Translation

Better Localization has recently published an interesting article on how to predict the productivity of an MT and post-editing process.

The article also deals by necessity with the translation quality level of machine translation, as the productivity that can be achieved during post-editing heavily depends on the quality of the MT output.

I have some issues with some of the things the article says (for instance that "testers and cannot give reliable quantitative results in a cost-effective manner.": I believe that the work Franco and I have done on translation quality measurement is cost effective and could be adapted to MT as well, but the article is very well worth reading, and suggests a novel approach that looks promising.

Article for Multilingual: Translation Quality Measurement and the Translation Quality Index

That is the tentative title for the article Franco and I are committed to write for a forthcoming issue of Multilingual devoted to Translation Quality. Jeff Allen is coordinating the issue, will write a companion article and, as I understand, has another two articles being written by others.

Now the hard part: writing the article itself.

Language Standards For Global Business

Interesting press release about the first language standards summit, which was held a few days ago in Berlin.

Practitioners, corporate planners, suppliers, and academics came together and actively shared their views on what they needed in language technology and process standards.

Among the issues tackled were language standards and translation quality.

The next summit is scheduled for May 29 and 30, in Barcelona.

Translation Quality Measurement in Practice

You can download our latest presentation from our website TranslationQuality.com.
We gave the presentation on Saturday at the 46th ATA Conference, in Seattler.
The presentation went well: I was nervous, and I think it showed, but Franco handled his part with ease, and I think that there was a fair amount of interest.

TranslationQuality.com Now Up

Our new web site, devoted to translation quality issues (at present mainly our work on the measurement of translation quality), is now up.

You can visit it at TranslationQuality.com.

Another blog of ours that you may find interesting is About Translation, devoted to information, news and opinions about professional translations.

Translation Quality Measurement in Practice

This is an abstract for the presentation we plan to give at the next ATA conference.

In previous years, we proposed a theoretical framework for assessing and measuring translation quality, explaining how the Translation Quality Index (TQI) can be a reliable indicator of translation quality and serve as the foundation for process improvement.
This year we will show how we have turned our theory into practice. In particular, we'll discuss how translation quality measurement is being implemented worldwide at a leading translation company, and we'll share our experience and real-life examples.

The presentation will include a demonstration of tools and services that they developed to help measure translation quality.