A "crawler" is a generic term for any program, such as a robot or spider, that is used to automatically discover and scan websites by following links from one webpage to another. Smartling's main crawler is called SmartlingBot. This table lists information about the Smartling crawlers you may find in your logs.
| Crawler | Purpose | Example User Agent String |
| SmartlingBot |
The SmartlingBot is used by Smartling's Context Service to provide Visual Context of content for translator's in the CAT Tool.
An example use case for the crawler is providing Visual Context for any of the Connector integrations, such as the Zendesk Connector. The Zendesk Connector requests as the SmartlingBot, and as a result, only the SmartlingBot needs to be whitelisted.
The SmartlingBot does not respect robots.txt rules because it grabs context only once for each explicitly requested asset. |
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36 SmartlingBot/W.X zendesk-connector/Y.Z.
"WX", "YZ" will be replaced with current version numbers in the logs |