Edit Effort Estimation for Machine Translation

 

What is Edit Effort Estimation for Machine Translation?

As part of Smartling’s AI Toolkit, Edit Effort Estimation can help you achieve cost savings for content that goes through a Machine Translation step, followed by a step for human Post-Edit, Edit or Review ("MTPE").

Edit Effort Estimation uses AI to predict the estimated translator effort for each string that goes through a Machine Translation step. This means that Smartling can estimate how many edits a linguist will need to make in the post-editing step before the string can be published.

Based on this analysis, human validation can be utilized in the most efficient way, for example by setting up a dynamic workflow based on the predicted edit effort.

How are strings assessed?

There are two options available for Edit Effort Estimation (EEE) assessment:

1. Standard (LLM)
2. Fine-tuned model (XLMR)

Standard

Smartling uses a Large Language Model to assess the Machine Translation output based on four criteria:

1. Grammatical correctness
2. Fluency
3. Semantic coherence
4. Lexical accuracy

The Machine Translation output is also checked against your Linguistic Assets: 

• The Machine Translation output is compared to matches from your Translation Memory that may be available.
• The Machine Translation output is evaluated based on the criteria established in your Quality Check Profile. If any Quality Check errors are detected, it is assumed that a high level of human effort will be needed during the editing process.
  If Glossary Compliance checks are enabled, Edit Effort Estimation also verifies if any available glossary terms were applied in the Machine Translation output.
• If a Style Guide using the Smartling template is available, the Machine Translation output is compared against elements such as the “Do not translate” language conventions. For optimal performance, it is also recommended to specify the industry (domain) of your organization or brand in the Style Guide.

Fine-tuned model

Smartling offers the option to purchase a custom fine-tuned model for Edit Effort Estimation (EEE) assessments. This provides higher-quality estimation results, allowing the three levels of effort to be assigned with much greater accuracy, as the model is trained on your specific data. Instead of relying on a large language model (LLM) for EEE, fine-tuned models use machine learning through Cross-lingual Language Modeling (XLMR), similar to training a custom MT engine.

However, unlike custom MT engine training, where you need a different model for each locale pair, you only need one fine-tuned model for EEE, which will be used for all your locale pairs based on one source language and your target language(s). 

Once a fine-tuned model has been created, you can select it for EEE within the workflow step configuration. 

If you are interested in using fine-tuned models for EEE, please reach out to your Customer Success Manager to discuss this option.

Edit Effort Estimation levels 

During the Machine Translation step, each string gets assigned a label, based on the predicted level of edit effort required.

• Level 1:
  No or very little edit effort required.
  For strings with the label "Level 1", the human post-edit step can potentially be skipped as part of a Dynamic Workflow.
• Level 2:
  Low edit effort required.
  These strings are often understandable in the target language, but require some validation or light editing to ensure an idiomatic translation.
• Level 3:
  Medium or high edit effort required.
  For strings with the label "Level 3", we recommend never to skip the human post-editing step. These strings typically require more extensive human validation and editing.
  
  

Where are Edit Effort Estimation levels displayed?

Once your content has gone through Edit Effort Estimation, the level assigned to each string can be used to analyze and take action on your MTPE workflows.

Edit Effort Estimation in the Strings View

The level assigned to each machine translated string can be checked in the Strings View:

• In Smartling's default view, the Edit Effort Estimation level is displayed in the Translation column, below the saved translation.
• Alternatively, the Edit Effort Estimation level can be displayed as a separate column in the Strings View, by creating a Custom View.

The Edit Effort Estimation filter allows you to filter the Strings View by the predicted effort level. 

Edit Effort Estimation in the Word Count Report

To help you achieve cost savings for content going through a human post-editing step, your Word Count Report displays the Edit Effort Estimation levels for content that was submitted from a Machine Translation step.

Edit Effort Estimation Level Discounts in your Fuzzy Match Profile

Based on your vendor agreement with your Language Services Provider, lower post-editing rates may apply for content with a low estimated level of edit effort.

Once the AI Toolkit has been enabled for your Smartling account, you can create a Fuzzy Match Profile that takes into account potential discounts based on Edit Effort Estimation levels. You can enter the payable rate percentage for each level, as per your agreement with your translation vendor.

As soon as the AI Toolkit has been enabled for your account, you also have the option to update your existing Fuzzy Match Profiles with Edit Effort Estimation Level Discounts.* Please note that this can be done only once. As soon as Edit Effort Estimation Level Discounts have been populated and saved for your existing Fuzzy Match Profile, it is not possible to edit them at a later stage. To avoid altering Fuzzy Match Profiles during an ongoing billing period, a new profile needs to be created if Edit Effort Estimation Level Discounts need to be amended at a later point.

In order for Edit Effort Estimation Level Discounts to take effect, ensure to apply your Fuzzy Match Profile also to the post-translation step.

*It is not possible to add Edit Effort Estimation levels to the system-generated default profiles, 'Default' and 'No Discount'. If you need to use one of these default profiles as your account default or for certain workflows and add Edit Effort Estimation discounts, you will need to create a copy of the system-generated profile and use that one to add Edit Effort Estimation discounts.

 

Edit Effort Estimation in Cost Estimates

Edit Effort Estimation levels and discounts are shown in the cost estimate for your translation Job:

• if Edit Effort Estimation is enabled for the workflow used to generate the cost estimate, and 
• if Edit Effort Estimation Level Discounts are set up in the associated Fuzzy Match Profile.

In order for Edit Effort Estimation levels to be displayed, the cost estimate is refreshed automatically once all content has gone through the Edit Effort Estimation process in the Machine Translation step.

 

Edit Effort Estimation in the CAT Tool

The linguists working on the post-editing step can see the Edit Effort Level assigned to each string in the CAT Tool. This helps them gauge the effort expected to be required for each string during the editing process. 

Info: For strings with a fuzzy match discount, only the fuzzy match percentage is displayed in the Word Count Report and in the CAT Tool. The Edit Effort Estimation level is not displayed in the Word Count Report or in the CAT Tool if the available Translation Memory match qualifies for a discount, as per your Fuzzy Match Profile.

 

Content routing based on Edit Effort Estimation levels

When creating a Dynamic Workflow, Edit Effort Estimation levels can be used as a criteria to route content to different workflow steps:

• Edit Effort Estimation levels are applied during the Machine Translation step. 
• A Dynamic Workflow with a post-translation Decision step can be created to route the machine translated content based on the estimated level of effort.
• Based on the level of edit effort, the strings can be sent to different branches in the workflow.
  For example, strings with the label “Level 1”, which are unlikely to require any human intervention, could be sent directly to the Published step, while strings with the label “Level 2” are likely to require some light editing and could be sent to a Review step for final touches. Strings with the label “Level 3”, which are very likely to need a medium or high level of editing, could be sent to a Post-Edit step.

Example workflow:

Routing content to the right step based on their estimated level of edit effort can help save translation costs and utilize human editing in the most efficient way. 

How to enable Edit Effort Estimation 

Edit Effort Estimation is only available as part of Smartling’s AI Toolkit. To learn more or to enable the AI Toolkit for your account, please reach out to your Smartling Customer Success Manager.

Once the AI Toolkit has been activated for your account, Edit Effort Estimation then needs to be enabled on a workflow step level:

1. In your Smartling project, navigate to Settings > Workflows.
2. On a workflow that uses Machine Translation, click Manage Step on the Translation step.
   
   
   
3. Ensure that the toggle to Enable Edit Effort Estimation is switched On.
   By default, this setting is disabled.
   

   Tip: If this setting is not visible for your workflow, please ensure to select an MT Profile in the "Translation Provider" dropdown first.

4. [Optional Add-On] If you have purchased a fine-tuned model for EEE, select your custom model from the dropdown menu. 
   

Supported workflows

We would recommend enabling Edit Effort Estimation only on MTPE workflows, i.e. workflows with a Machine Translation step, which is followed by a step for human Post-Edit, Edit or Review.

Edit Effort Estimation is not available for workflows managed by Smartling Language Services, including AI-Powered Human Translation. 

Important considerations

SmartMatched strings are not evaluated 

If a string uses SmartMatch, no Edit Effort Estimation level is assigned, as the string bypasses the Machine Translation step.

Supported languages

While Edit Effort Estimation is available for all locales supported by Smartling, we would recommend using it for the most common, high-resource languages for the best results.