When used as a translation provider, Large Language Models (LLMs) are prone to hallucinations, which can produce nonsensical, incorrect, or inconsistent translations.

To help prevent publishing problematic translations, Smartling's hallucination detection feature automatically flags potential hallucinations, allowing you to route affected strings to an alternative provider or workflow.

How it works

Smartling provides automated hallucination detection to help identify potentially problematic translations. This service uses a non-LLM Google embedding model (Vertex AI) that evaluates the semantic similarity between the source and translation to determine whether a translation is a potential hallucination.

If you prefer not to send data to Google, an alternative option using a different embedding model, LaBSE, is available. Please contact your Customer Success Manager to enable this alternative option.

Hallucination detection in a workflow

When an LLM is used as the translation provider in a workflow and a potential hallucination is detected, Smartling falls back to the alternative MT Profile configured for the workflow.

If the alternative provider also returns a potential hallucination, or if no alternative provider is configured, Smartling flags the string by opening a translation issue. To resolve the issue, you can:

• Move the string into a workflow that uses human translation or a different MT provider, or
• Manually enter a translation.

As with any automated translation, we recommend keeping a human in the loop to validate and edit translations as needed.

Hallucination detection for MT integrations and the CAT Tool

Hallucination detection also runs when an LLM is used as the translation provider for Smartling's instant MT integrations or for MT suggestions in the CAT Tool.

For integrations using Smartling's MT API, the translation is still returned when a potential hallucination is detected, but a validation error in the response alerts you to the issue. You can then decide how to handle the flagged translation.

Example:

Disabling hallucination detection

Hallucination detection is enabled by default when an LLM is used as your primary translation provider. To disable it for a specific LLM Profile:

1. From the AI Hub, navigate to the Profiles page.
2. Click the name of the LLM Profile you want to update.
3. Click the arrow to expand the Parameter Details section.
   
4. Select the Disable hallucination detection checkbox.
   
5. Click Next and then Save.