Select models using benchmarks
Unit 3 of 7 · Module 2: Select, deploy, and evaluate Microsoft Foundry models
Before you start
Short explainers for ideas this lesson builds on. Watch these first if they're new to you.
Go deeper
Unlocks when you've watched the lesson.
Read this lesson
Two ways in
Before you commit to a model, you'll want proof of how it performs. Benchmarks are that proof: objective, measurable scores you can compare, grouped in the Foundry portal into quality, safety, cost, and performance. You can reach them from two places. In the model catalog, the model leaderboard ranks a curated set of popular language models, so the top performers for a metric or a scenario jump out. At a glance, it lists the best models for quality, safety, estimated cost, and throughput. Or, if you already have one model in mind, open its model card and go to the Benchmarks tab. There you'll see how it scores across metrics and datasets, with comparison charts that set it beside similar models.
Quality
Quality benchmarks ask a simple question: are the model's answers accurate, coherent, and right for the context? They're run on public datasets with standardized methods, so every model takes the same test. The quality index is the headline number: it averages accuracy across datasets that cover reasoning, knowledge, question answering, math, and coding. The higher it is, the stronger the model. Among the datasets: BIG-Bench Hard, for reasoning. MuSR, for reasoning through several steps. GPQA, with graduate-level science questions. ChemBench, for chemistry. MBPP plus, for writing code. MMLU-Pro, for broad general knowledge. And tau two telecom, for picking the right tool as an agent. Every score is normalized to a range from zero to one, where higher is better.
Safety
Safety benchmarks check whether a model can be pushed into saying something harmful, biased, or inappropriate. That's crucial for anything your users will see, and even more so in customer-facing apps, or regulated industries like banking and healthcare. Foundry measures safety from a few angles. HarmBench tests how well a model resists requests for unsafe content. It reports an attack success rate, meaning how often those attempts worked, so here, lower is safer. Its attacks come in three flavors: plainly harmful requests, like cybercrime or illegal activity; requests that turn harmful in context, like misinformation, harassment, or bullying; and copyright violations. ToxiGen checks how well a model spots hate speech, including the sneaky, implicit kind that's written to slip past filters. It's scored with an F1 score, where higher means better detection. And WMDP, short for the Weapons of Mass Destruction Proxy benchmark, probes what a model knows about biosecurity, cybersecurity, and chemical security. Here a higher score means more knowledge of potentially dangerous capabilities, so treat it as a warning light, not a trophy.
Cost
Cost benchmarks help you balance quality against your budget. They show prices for serverless API deployments and for Azure OpenAI models. Input cost is what you pay for every million tokens you send to the model. Output cost is what you pay for every million tokens it writes back. The estimated cost rolls both into a single number, assuming a typical mix of three input tokens for each output token; Foundry is also starting to show what each model actually cost to run through the benchmark tests. Lower means cheaper to run. What you're after is a model that's good enough for the job, at a price your usage can afford: a chatbot handling ten thousand chats a day will feel every fraction of a cent.
Performance
Performance benchmarks measure how quickly and efficiently a model responds, which matters most for real-time apps. Latency is about waiting: the mean is the average number of seconds per request, and percentiles show the slow cases, starting with P50, the median, where half of requests finish faster, then P90, P95, and P99. Time to first token is how long you wait for the first word to appear when the reply streams in. Throughput is about speed once it's going: generated tokens per second, total tokens per second counting input and output, and the time between tokens. On the leaderboard, two numbers sum it up: mean time to first token, where lower is better, and mean generated tokens per second, where higher is better. A chat app wants both, high throughput and low latency, so replies feel instant. A batch job that runs overnight can trade some speed for a lower price.
Choosing with the leaderboard
Back on the model leaderboard, you can sort by quality, safety, estimated cost, and throughput. Scenario leaderboards go further, ranking models for reasoning, coding, math, question answering, or groundedness. If your app fits one of those, like our support chatbot, where groundedness matters a lot, start with that ranking, not just the overall quality index. Trade-off charts plot two metrics at once: quality against cost, throughput, or safety, chosen from a dropdown. Models near the top-right corner do well on both. And sometimes a model that's a little less accurate, but much faster or cheaper, is the smarter pick. Last, side-by-side comparison: check the boxes next to two or three models, then choose Compare. You'll get their benchmarks, model details like context window, training data, and supported languages, plus their supported endpoints and features, like function calling, structured output, and vision.
Facts from Microsoft's documentation: Model benchmarks and leaderboards in Microsoft Foundry (CC BY 4.0), rewritten in Loop's own words and examples. Narration, animation, and quiz by Loop. Microsoft, Azure, and Microsoft Foundry are trademarks of Microsoft. Loop is independent and not affiliated with or endorsed by Microsoft.