Ground your model with Retrieval Augmented Generation
Unit 3 of 7 · Module 5: Optimize generative AI model performance with Microsoft Foundry
Before you start
Short explainers for ideas this lesson builds on. Watch these first if they're new to you.
Go deeper
Unlocks when you've watched the lesson.
Read this lesson
Grounding
Prompt engineering shapes how a model answers, but it can't teach the model facts it never learned. Its training data stops at a cutoff date, and it never included your organization's private information. So when the model is missing the context it needs, it can still produce an answer that sounds convincing, and is simply wrong. The cure is grounding: handing the model relevant, factual material to build its answer on. The most common way to do that is retrieval augmented generation, or RAG. Think of the difference between a closed-book exam and an open-book one: without grounding, the model answers from memory alone. The answer can read beautifully, and still be inaccurate, or completely made up. Ask which hotels the agency offers in Paris, and it may hand you fictional hotel names. Ground the prompt instead, sending trusted, relevant data along with the question, and the model answers from that data, like a student allowed to open the textbook. Give it the agency's real hotel catalog, and back come real hotel names, prices, and availability. Grounding makes answers more accurate, by tying the model to information that's specific, current, and relevant.
Retrieve, augment, generate
RAG finds relevant information in a data source, and adds it to the prompt before the model answers. It happens in three steps. Retrieve: search a data source for anything related to the user's question. Augment: tuck what you found into the prompt, as context. Generate: send that fuller prompt to the model, and get back a grounded answer. The model now works from fresh, relevant facts, not just what it remembers from training. To find the right information fast, RAG relies on embeddings and vector search. An embedding turns text into a vector: a list of decimal numbers that captures what the text means. You get embeddings by sending your content to an embedding model, like an Azure OpenAI embedding model in Microsoft Foundry. Take two sentences: the puppy dozed on the sofa, and a young dog napped on the couch. Hardly a word in common, but the same idea, so their vectors land close together, like two pins on a map of meaning. Cosine similarity measures how close they are, using the angle between the two vectors: the nearer it gets to one, the more alike they are. That's how you find the right documents, even when the wording doesn't match.
Azure AI Search
In Microsoft Foundry, the retrieval step of RAG is handled by Azure AI Search. You bring your data, build a searchable index, and query it. First, add your data to Foundry: from Azure Blob Storage, Azure Data Lake Storage Gen2, or Microsoft OneLake, or by uploading files directly. Next, build an index, with an embedding model turning your content into vectors. That index lives in Azure AI Search. Then, when someone asks a question, it's turned into an embedding too, the most similar content is found, and those results come back. Azure AI Search offers several ways to search. Keyword search looks for the exact words. Semantic ranking uses a language model to re-order results by how well they answer the question, not just by matching words. Vector search compares embeddings to find similar content. And hybrid search runs keyword and vector search together and merges the results; add semantic ranking on top, and you get the most accurate setup, the recommended choice for generative AI apps. Once your index is ready, the Foundry SDK, the Azure AI Projects package, gives you an authenticated OpenAI client from your project. Call responses dot create with a system message carrying the retrieved context, plus the user's question: which hotels do you offer in Paris? Since the search results ride along in the system message, the answer is grounded in your real data.
When to use RAG
RAG shines in four situations. When the model needs knowledge from your own field that it was never trained on, like a product catalog, policy documents, or an internal wiki. When information changes often, like stock levels, prices, or news, because RAG fetches current data at the moment of the question, with no retraining. When getting the facts right really matters. And when you need information from after the model's training cutoff. For the travel agency, RAG means customers can ask about particular hotels, destinations, and booking rules, and every answer comes from the agency's real catalog. And if you're building agents that need grounded knowledge, but you'd rather not run your own search setup, look at Foundry IQ, a managed knowledge layer: you build a knowledge base from your data sources, and your agents search it for you.
Facts from Microsoft's documentation: Retrieval augmented generation (RAG) and indexes in Microsoft Foundry; Vector search overview (Azure AI Search); Hybrid search overview (Azure AI Search) (CC BY 4.0), rewritten in Loop's own words and examples. Narration, animation, and quiz by Loop. The coding lab is based on code samples in Microsoft's Azure AI documentation (MicrosoftDocs/azure-ai-docs, MIT License; notice). Microsoft, Azure, and Microsoft Foundry are trademarks of Microsoft. Loop is independent and not affiliated with or endorsed by Microsoft.