Generate responses with the Responses API
Unit 4 of 7 · Module 3: Develop a generative AI chat app with Microsoft Foundry
Before you start
Short explainers for ideas this lesson builds on. Watch these first if they're new to you.
Go deeper
Unlocks when you've watched the lesson.
Read this lesson
One API for conversations
The OpenAI Responses API brings together two previously separate APIs, ChatCompletions and Assistants, in one unified experience. It's stateful, handling multi-turn conversations, which makes it ideal for conversational apps, and you reach it through an OpenAI-compatible client from either the Foundry SDK or the OpenAI SDK. Its advantages: it keeps conversation context across turns; it combines the chat completions and Assistants patterns; it works with Foundry direct models, not just Azure OpenAI models; and it's simple to integrate. It's the recommended way to generate responses in Foundry apps, replacing ChatCompletions for most scenarios.
Your first response
To generate a response, call responses dot create, passing your model deployment name and an input: your prompt. Then read the output text. The response object also carries an ID that uniquely identifies it, a status, like completed, usage information with input, output, and total tokens, and the model that generated it. Add instructions, often called a system prompt, to guide the model's behavior. And control generation with parameters: temperature, from zero to two, where higher is more creative and varied; max output tokens, to limit response length; and top-p, an alternative to temperature for controlling randomness. When you connect to a project endpoint with the Foundry SDK or an AzureOpenAI client, the same call works with Foundry direct models too, like Microsoft Phi or DeepSeek.
Keeping the thread
For a conversation, pass the previous response's ID as previous response ID on the next call. In a real app, that's a loop: the user types a message, you create a response that includes the last response's ID, print the output, and keep the new ID for the next turn. Each turn, the model receives your instructions, the new input, and the previous response, so every answer is grounded in what came before. Alternatively, manage the history yourself: keep a list of messages, add each response's output and each new user message, and pass the whole list as the input. That gives you more control: you can choose which messages to include, prune the conversation to stay within token limits, and store and restore history from a database. And because the API keeps response history, you can retrieve any earlier response by its ID.
Watch the tokens
There's a catch. Keeping history can increase token usage. On every request, the active context window can include your system instructions and safety rules, the current prompt, the conversation history, tool schemas, tool outputs like search results or code interpreter output, and retrieved memory or documents from RAG or file search. All of it is concatenated, tokenized, and sent to the model together, every time. The SDK helps you manage state, but it doesn't make tokens cheaper.
Keep it responsive
Responses can take a while, depending on the model, the size of the context window, and the prompt, and users get frustrated if your app seems to freeze. For long responses, set stream to true, and you receive output incrementally, so users see text as it arrives. Print the text delta events as they come, and when the response completed event arrives, grab the response ID for your conversation history. For busy apps, there's an async client, AsyncOpenAI, that doesn't sit idle while it waits for the network. The speed-up only comes when you send several requests at the same time, for example with asyncio gather, so their waits overlap. One request on its own takes just as long as before. And async streaming works the same way.
Facts from Microsoft's documentation: Use the Azure OpenAI Responses API; Get started with Microsoft Foundry SDKs and endpoints; How to use structured outputs with Azure OpenAI in Microsoft Foundry Models (CC BY 4.0), rewritten in Loop's own words and examples. Narration, animation, and quiz by Loop. The coding lab is based on code samples in Microsoft's Azure AI documentation (MicrosoftDocs/azure-ai-docs, MIT License; notice). Microsoft, Azure, and Microsoft Foundry are trademarks of Microsoft. Loop is independent and not affiliated with or endorsed by Microsoft.