Capturing Visual Context

Context from Video

This article is for Account Owners and Project Managers.
It is currently in public beta available to all customers.  Uploading is only possible using the API. Uploading via the dashboard will be supported in the future.

This article describes using video screen recordings to contextualize mobile, desktop and embedded apps.  For subtitles and video see the video subtitle article.

Not all application user interfaces are rendered as HTML. Native mobile and desktop apps often do not have any HTML version at all. Smartling supports Image Context (screenshots) when HTML is not available.  How can you generate and upload them when your application has lots of screens and user interfaces where the strings can display? How can you do this at scale and as automatically as possible?

Nowadays it's fairly easy to generate a high quality video screen capture - it's built into iOS and MacOS, and there are a number of apps available for Windows and Android.  Isn't recording a video while running your app to display the various UIs and screens much easier than stopping to take screenshots as you navigate from screen to screen? Smartling's Context from Video feature accepts a video to be used for context. It automatically processes the video to create Image Context (screenshots) and also connects the strings from your project to the images that are extracted from the video. Translators will see the context for those strings as screenshots (still images) in the CAT tool.

NOTE: Translators will not see the video in the CAT Tool, just the extracted screenshots.

Since the process produces standard Image Context resources, all the features for Image Context are available after your video is processed. You can edit the Image Context items to add or remove the strings associated with that image as needed to correct for anything the automation didn't get quite right.


How to generate great visual context for your native apps:

  1. Using the screen recording app of your choice, step through your application while recording the entire screen or the app.
    * please use test/dummy data to avoid exposing sensitive or confidential information
  2. Post the video at a publicly available URL 
  3. Use the API to upload the video to the correct Smartling project (dashboard UIs are coming in the future).

After processing is complete a set of Image Contexts will be in your project. They will use the same url to identify the source file and a unique title like "Frame 1 – 01:10.300".

These will have strings matched using OCR. You can then update the strings as normal via Image Context API or dashboard features.

You can review the strings in one of two ways:

  1. On the Context tab for your project all the images will use the URL and a time stamp; enter the original URL in the context Title/URL search field to narrow down the context assets to just those of the video.
  2. In the Strings tab for your project enter the original URL in the Context Search filter (do *not* click "exact") to narrow down to the strings those that have been associated with Image Context from the video.

Technical Requirements for video files used for video context

The video must be posted at a publicly available URL in the MP4 format (no login should be required to access the video file). Youtube or other hosted solutions are not supported.

Video length and resolution

Up to 5 minutes for 2000x1100px (or 1100x2000px) 

Up to 2 minutes for 3200x2100px (or 2100x3200px) 

Video file size 

Up to 500Mb


Up to 60fps


API reference

You will make a single POST API call with the video’s URL to the to the /context-api/v2/projects/{projectId}/contexts/upload-and-match-async API endpoint. The API response will contain an matchId you can use to check on the video processing progress and get results using the /context-api/v2/projects/{projectId}/match/{matchId} endpoint.

Example POST call

POST  /context-api/v2/projects/{projectId}/contexts/upload-and-match-async

Content-Type: multipart/form-data; boundary


Content-Disposition: form-data; name="name" 


If there are no strings matched to the video then the original video will be deleted and screenshots will not be created.
/match/{matchId} endpoint will return empty bindings in this case:
"bindings": []

Was this article helpful?