Detect language
Unit 3 of 7 · Module 16: Analyze text with Azure Language in Foundry Tools
Before you start
Short explainers for ideas this lesson builds on. Watch these first if they're new to you.
Go deeper
Unlocks when you've watched the lesson.
Read this lesson
A tasting for messages
Language detection works a bit like a wine taster naming where a glass comes from. You hand it a piece of text, and it tells you three things. The language's name, like Portuguese. Its short code from the ISO six three nine one list, like p t. And a confidence score between zero and one, where numbers near one mean the model is very sure. Wayfarer uses it on guest reviews, where nobody knows the language in advance. It's also handy in a chat app: detect the guest's first message, then answer in their language. The message, can I move my booking to Friday, is labelled English. The same question in Portuguese is labelled Portuguese. Both get high scores, because short, clear sentences are easy to place.
A tray of documents
Each piece of text you send is called a document. You send them together as a list, and every document gets an ID. You get back one result per document, in the same order. In Python, you call detect underscore language on the client and pass your list. Then you loop over the response and print each document's ID, and its primary language: the name, the ISO code, and the confidence score. There are limits. Each document has to stay under five thousand one hundred and twenty characters, and one request can carry up to a thousand documents. If one document is too long, only that one comes back as an error. The others still get their results.
Blends and blanks
Some messages mix languages. Breakfast on the terrace was lovely, obrigado a todos! Here the service picks the language with the biggest share of the text, English, but gives it a lower score, because the call was closer. Other text can't be placed at all. A booking reference made only of digits, or text garbled because a file was read with the wrong character encoding, comes back as Unknown, for both the name and the code, with a score of zero.
Giving a hint
Very short text can be ambiguous. The word hotel is spelled the same in English, Portuguese, Spanish, French and German. By default, the service assumes the text comes from the United States. If you know the guest wrote from Portugal, you pass a country hint of PT, and the service leans toward Portuguese. For some languages, it also tells you which script the text is written in, like Latin or Cyrillic, with a four letter code. And if you'd like to try all this before writing code, the Foundry portal has a language detection playground.
Facts from Microsoft's documentation: What is language detection in Azure Language?; How to perform language detection; Service limits for Azure Language (CC BY 4.0), rewritten in Loop's own words and examples. Narration, animation, and quiz by Loop. Microsoft, Azure, and Microsoft Foundry are trademarks of Microsoft. Loop is independent and not affiliated with or endorsed by Microsoft.