The Nonsense-Free Guide to Picking Your AI Model

What this course covers

This course is about choosing an AI model for real work. You will compare models on quality, speed, cost and risk. You will also learn to read benchmark results with care. Chapter 1 explains why the choice is harder than it looks. This first lesson starts with the question people ask first: which model is best?

Here is one way that question arrives. Someone on a product team posts a link to a public leaderboard in the team chat. The message says, "This model is ranked first. Why aren't we using it?" A few people reply with a thumbs-up. The thread moves on without anyone asking what the team needs the model to do.

As asked, that question has no useful answer. This lesson shows why. It also shows how to run a small test that gives a better answer for your own task.

"Best" depends on the task

A model is not good or bad in general. It is good or bad at a specific task, on specific inputs, within specific limits. Change any one of those and the ranking can change.

Take two jobs at the same online shop.

• A chat assistant answers shipping questions on the website while the customer waits. A slow answer feels broken, so speed matters a lot. Each answer is short.
• A nightly job tags 20,000 product reviews by topic for a weekly report. It runs overnight, so a few extra seconds per review do not matter. The volume is high, so the cost of each call adds up fast.

The same model can be a sound pick for one job and a poor pick for the other. A large, slower model may write better chat answers. A small, cheap model may tag reviews just as well for far less money. You find out which is which by testing each job on its real inputs.

So the useful question is not "which model is best?" It is "which model is good enough for this task, on our data, within our limits?" That question has three inputs:

• Your task: what the model must produce, and who uses the output.
• Your data: the real inputs, with their length, mess and quirks.
• Your constraints: budget, how long people can wait, where your data may go, and the tools you already run.

Without those three, "best" has no meaning you can act on.

What public benchmarks measure

A benchmark is a fixed set of test tasks with a scoring method. Every model takes the same test, so the scores can be compared.

You will meet a few kinds:

• Academic benchmarks. MMLU is one example. It asks multiple-choice questions across many school and professional subjects. Others test maths word problems or short coding tasks. The answers are known, so scoring is automatic.
• Preference leaderboards. People see two unnamed answers to the same prompt and vote for the one they prefer. The votes become a ranking.
• Domain benchmarks. These test one area in depth, such as fixing real code or answering questions about long documents.

These tests are useful. A strong result on a hard reasoning test tells you the model can reason well in that format. A ranking can help you choose a few models to try. Treat benchmarks as a first filter for a shortlist, not as the decision.

What public benchmarks miss

A benchmark measures a model on someone else's task. Your task differs in ways the score cannot show.

• Your inputs. MMLU's questions are short multiple-choice items with one correct answer. Your inputs might be long email threads, scanned forms or notes full of typos and shorthand.
• Your idea of a good answer. A multiple-choice test checks one letter. Your task might need a summary that keeps every account number exact and adds no facts.
• Speed under your load. A score says nothing about how long an answer takes when many requests arrive at once.
• Cost at your volume. A model that wins by a small margin can cost several times more per request.
• Your data rules. A score cannot tell you whether a provider may store your inputs, or for how long.

Two more limits matter. The first is contamination. If test questions leaked into a model's training data, it can score well by recall rather than skill. The second is timing. Providers update their models, so an old score may describe behaviour that has since changed.

A common misconception is that a higher score means a better model for your work. A higher score means the model did better on that test. Whether that carries over to your task is something you test. A model that ranks lower on a broad leaderboard can beat the leader on your own examples. The reverse can happen too.

Define "good enough" before you test

Before you look at any results, write down what a passing model must do. Set a bar for each of three things:

• Quality: how many answers must be correct, and which errors are not acceptable at all.
• Latency: how long a person or system can wait for an answer. Latency is a spread, not one number, so set the bar on the slow end, such as the 95th percentile, under realistic load.
• Cost: what you can spend per request, or per month at the volume you expect.

Write each bar so that you can check it. "Accurate" is not a bar. "At least 46 of 50 summaries rated usable, and zero with an invented figure" is a bar. These written bars are your acceptance criteria.

Why write them first? If you set the bar after you see the results, it is easy to set it where your favourite model landed. A bar written first turns the test into a pass-or-fail check.

Writing bars also forces a useful talk with the people who own the budget and the users. Someone may ask for top quality, the fastest answers and the lowest cost at once. Ask them to rank the three. That ranking is the real requirement.

A worked example: summarising support tickets

Illustrative example: an operations analyst at a 300-person software company wants AI summaries of support tickets. When a ticket moves from the support team to an engineer, the engineer should get a five-line summary. The team hands off about 4,000 tickets a month. Tickets contain customer data. Company policy says the provider must not use the inputs for training.

She writes her bars first.

Area	Bar	How she checks it
Quality	46 of 50 summaries usable; zero with a wrong account ID or invented step	Two engineers score each summary against a short checklist
Latency	Each summary ready within 15 seconds of the handoff	Time each of the 50 calls
Cost	Monthly spend at 4,000 tickets fits the support tools budget	Count input and output tokens per call separately, and multiply each by that provider's current rate
Data	Provider terms say inputs are not used for training	Read the terms and confirm with the security team

The data bar works as a gate. A model that fails it is out before any testing starts.

She builds a shortlist of three models from public rankings and provider documents. Then she runs the test on 50 real tickets.

• The top-ranked model scores 49 of 50 and is fast enough. Its monthly cost at 4,000 tickets is over budget, so it fails the cost bar.
• A mid-sized model scores 47 of 50 with no wrong IDs. It is fast enough and fits the budget.
• The smallest model is the fastest and cheapest. It invents a troubleshooting step in three summaries, so it fails the quality bar.

She picks the mid-sized model. It does not top any leaderboard. It is the model that clears every bar she wrote before she saw a single result.

Run a small evaluation on your own examples

An evaluation, or "eval", is a repeatable test of models on your own examples. Here is a small version of one.

1. Collect 30 to 50 real examples. Take them from your actual work, with private data removed if needed. Include the hard cases: long inputs, messy formats and requests that caused trouble before.
2. Write down what a good answer contains. For each example, list the facts the answer must include. Or keep a reference answer written by someone who knows the work.
3. Write a short scoring checklist. Use three to five yes-or-no checks, such as "keeps every figure exact" and "adds no facts". Yes-or-no checks are easier to agree on than a score out of ten.
4. Run every model with the same prompt and settings. Record each answer, the time it took and the tokens it used.
5. Score blind. Hide which model wrote which answer, so a liking for one brand cannot creep in. If two people score, compare their marks on ten answers. If they disagree a lot, sharpen the checklist wording.
6. Compare against your bars. Drop any model that fails one. Among the rest, pick the one that best fits your top priority.

Thirty to fifty examples is a first check. It can show a clear gap between models. It can show that a model fails on a type of input you care about. It cannot reliably separate two models whose scores sit a point or two apart. When the result is that close, add more examples, or decide on cost and speed.

Keep the examples and the checklist when you finish. They are now a test set you can run again.

Keep the choice easy to change

Picking a model is one decision. How tightly you tie your system to it is another. The first can be easy to reverse. The second can make a later switch slow and costly.

Your test set is half of what keeps the choice easy to reverse. Providers update and retire models, and new ones appear. When that happens, you run the same test on the new model. You can then see whether it clears the same bars before you switch.

The other half is how your system calls the model. Send every request through one small function in your code or workflow tool. Keep the model name in a setting, not typed into many places. A switch then starts with changing one setting and running your test again. Your prompts may still need some adjusting for the new model. Later lessons in this course cover version pinning and the cost of moving prompts between models.

What to do this week

Pick one task where you use an AI model now, or plan to.

• Write the task in one sentence, and name who reads the output.
• Write your bars for quality, latency and cost. Add any data rule that acts as a gate.
• Collect 30 real examples, with at least five hard ones.
• Run two models on them, score blind with your checklist, and see which one clears every bar, if either does.

That gives you an answer to "which model is best?" that holds for your task. It may differ from the leaderboard's answer. That difference is the reason to test.