AI for the Skeptic: Giving It a Second Chance

What this course covers, and where it starts

If you tried an AI tool and it let you down, this course does not ask you to forget that. It treats your doubt as evidence worth looking at. Over five chapters you will look at what AI tools do reliably, where they fail, and how to test a claim before you trust it.

This first chapter is about the doubt you already hold. Where did it come from, and how can you check it fairly?

This lesson does three things. It names real failures that give people good reason to doubt AI tools. It gives you four plain causes to sort your own past attempt. And it shows you how to run a small second test that you can check yourself.

It does not try to change your mind. A fair test can confirm your doubt as easily as it can weaken it.

Three failures that earned the doubt

Doubting AI tools is not the same as not understanding them. Here are three kinds of failure people report, and each has a real cause.

It misread a table. You pasted a spreadsheet into a chat tool and asked for a summary. The answer put a number in the wrong row, or gave a total that did not match. When a table is pasted as plain text, it can lose its column lines and headers. The tool then has to guess which number belongs to which label. A wrong guess can read just as confidently as a right one. Even with the headers kept, a chat tool that does not run code produces a total by prediction, not by arithmetic, so long sums and counts can be wrong.

It gave different answers to the same question. You asked the same thing twice and got two answers that did not agree. Chat tools built on large language models write a reply a small chunk of text at a time. Each chunk is chosen from likely options, and that choice can include some randomness. Earlier messages in the same chat can change the answer too, and so can updates to the model. So the same question can come back with a different answer. This means one answer does not show what the next will be.

It did not do what the vendor said. A demo showed a clean result. In your work, on your data, the result was worse. A demo is built from examples someone chose. A vendor has every reason to pick inputs the tool handles well. That is not always dishonest. But a demo shows the best case, not your case.

A fourth problem can show up in any of the three. A tool can state a figure, a quote or a source that does not exist. This is called a hallucination. The tool produces fluent text by predicting likely words. It does not check those words against a source. Tools that look things up first can still misquote what they found.

Each of these failures is real. If one happened to you, your doubt was a reasonable response to what you saw.

Earned doubt and reflexive doubt

There are two kinds of doubt, and the difference matters.

Earned doubt points at something specific. "I asked it to total our invoices by client, and two totals were wrong." You can check that sentence. Someone could repeat the test and see the same thing, or see something different.

Reflexive doubt is a general verdict. "AI doesn't work for our kind of work." It may have started from a real event. But it has lost its details, so no test can confirm or refute it.

Here is a misconception worth correcting: one failed attempt settles the question. It does not. A single attempt tests one task, one setup and one set of inputs on one day. It can show that the tool failed that test. It cannot show that the tool fails every version of the task.

The same rule cuts the other way. One good result does not show that a tool is reliable either. That is why this course asks for tests, not impressions.

Four ways a first attempt goes wrong

When you look back at a failed attempt, you can sort the cause into one or more of four groups. Each group points to a different change for next time.

Cause	What it looked like	What to change in a second test
Wrong task	Work the tool does unreliably, such as exact counts over many rows, or facts it could not see	Pick a narrower task with answers you can check
Missing context	A short prompt with no headers, no examples and no source material	Give the inputs a new colleague would need
No checking step	You read the output and trusted it, or caught errors by luck	Decide in advance how you will check each output
Inflated expectations	You were told it would take over a whole job, so a partial result felt like failure	Write down what "good enough" means before you start

Sometimes a tool fails even with a clear task, full inputs and a fair check. That is a finding about the tool, not about you.

An attempt can have more than one cause. A tool that misread a pasted table may also have been given no headers. The error may then have reached a customer because the totals were not compared.

Some causes sit with the tool, not with you. If a task needs facts the tool cannot reach, such as your company's current prices, a better prompt alone will not fix that. If you could have supplied the facts, the cause is missing context. If the tool could not use them even then, it is the wrong task. Naming this cause tells you to stop spending time on that task.

A worked example: the weekly late-shipment summary

Illustrative example. An operations analyst at a regional freight company builds a summary of late shipments each Monday. She starts from a spreadsheet export of about 200 rows. Each row is one late shipment, with columns for carrier, route, days late and a reason code.

Last year she pasted the rows into a chat tool. She asked which carriers had the highest number of late shipments that week. The answer looked clean and ranked three carriers with counts. Her manager compared it with the usual pivot table. Two counts were swapped and one carrier was missing. She stopped using the tool and told her team it "can't handle our data."

Now sort the attempt into the four causes.

• Wrong task, in part. Counting rows by carrier has one exact answer, and the pivot table already gives it. Asking a chat tool to count added risk and saved her nothing.
• Missing context. She pasted the rows without the header row and without the list of reason codes. The tool had to guess what each column meant.
• No checking step. The error was caught because her manager happened to compare. On a busy week, a wrong count could have gone to customers.
• Inflated expectations. The tool had been introduced to staff as something that "understands your data." That set a bar the setup could not meet.

Her doubt was earned. But "can't handle our data" is broader than what she tested. She tested one unlabeled paste, one question and one run, with no check.

A fair second test keeps the counting in the spreadsheet, where it is exact. It tests the step that takes her time: writing the short weekly note to account managers that explains the delays. She gives the tool the pivot table with its headers, plus the reason-code list. She uses five past weeks for which she already wrote the note herself.

Her pass rule, written before she starts, has two parts. Every number in the note must match the pivot table. And every reason must come from the code list. She also runs two of the weeks twice, to see how much the note changes.

If the test fails, she has a specific finding to report. If it passes, she has one small, checked use, and the check stays.

Why one bad try sticks, and why "it has improved" is not proof

The first-impression effect, also called the primacy effect, is an established idea in psychology. The first impression you form can pull later judgments toward it. A first bad experience with a tool can shape how you read every later report about it.

There is also a real cost to trying again. You spent hours on the first attempt, and maybe some standing with your team. It is reasonable to want a good reason before you spend more.

You will hear that AI tools "have gotten better." Treat that as a claim, not a reason. It may be true for some tasks and false for yours. The change that matters to you is one you can see on your own task, with your own inputs, checked against answers you already know.

When someone says a tool has improved, ask three questions. Improved at what? Measured how? On whose data? A specific answer gives you something to test. A vague answer is not evidence.

How to run a small, fair second test

A fair test is small enough to finish in an afternoon. It is also strict enough that the tool can fail it. Use these steps.

1. Pick one narrow task. Choose a task you do yourself and can check, such as pulling dates from contracts or drafting a reply from a policy page. Skip tasks where "good" is a matter of taste.
2. Gather cases with known answers. Use five to ten past examples where you already know the right output. Your own past work is the answer key.
3. Give the context a new colleague would need. Include headers, source documents, one example of good output that is not one of your test cases, and the rules you follow.
4. Write the pass rule before you run anything. For example: "Every date matches the contract, in all eight cases." Writing it first stops you from moving the bar after you see the results.
5. Check every output against the answer key. Mark each error and note what kind it is.
6. Run at least two cases twice. If the same input gives different answers, note how different. A task whose answer shifts between runs needs a check on every output.
7. Record the result in a few lines. Write down the task, the setup, the pass rule, the score and the date. That record is worth more than your memory of how the test felt.

Then decide. If the tool failed the pass rule, you now hold earned doubt about a specific task, with evidence. If it passed, you have a narrow use that keeps its checking step. Both are good outcomes for a fair test.

What to do this week

• Write one sentence that states your doubt as specifically as you can. Name the task, the tool if you remember it, and what went wrong.
• Sort that attempt into the four causes: wrong task, missing context, no checking step and inflated expectations. More than one may apply.
• Pick one narrow task for a second test, and collect five past cases with known answers.
• Write your pass rule before you open any AI tool.

Keep your sentence. The rest of this chapter builds on it. It covers where skepticism ends and cynicism begins, what a second look asks of you, and a written record of what you believe and doubt.