Opening soon
How LLMs Actually Work
Trace how a language model turns text into tokens, vectors and attention scores to predict the next token.
5 chapters · 24 lessons
Want a note when it opens?
Join the waitlist and we'll email you when the course is ready.
What you'll cover
The chapters
The plan for the full course. Lessons open when the course does.
- 01What large language models are and are not4 lessons
- 02How tokenization splits text into tokens5 lessons
- 03How embeddings place tokens in vector space5 lessons
- 04How the attention mechanism works5 lessons
- 05Multi-head attention and why it helps5 lessons
Self-check
How much do you already know?
5 quick questions. Pick an answer to see why it's right or wrong. Nothing is saved.
Question 1 of 5
What is the primary reason LLM output sounds confident even when the information is fabricated?
Question 2 of 5
What does the softmax function do in the context of LLM token prediction?
Question 3 of 5
What is the relationship between temperature and the probability distribution in token selection?
Question 4 of 5
What is distributional semantics?
Question 5 of 5
Which of the following best describes the relationship between vocabulary size and embedding table parameters?
In the news
Recent stories on this topic
From NewsPals, our sister newsroom.
- Human language may be shaping LLM physical reasoning, right into hydrogen bond traps (opens in new tab)
An arXiv study finds 28 models can rank molecular conformers well, until chemistry words tug their intuition off course.
- A Microsoft Researcher Built a Neural Network Out of Goats in Age of Empires 2. The Point Is Not What You Think. (opens in new tab)
Adrian de Wynter's absurdist experiment is the clearest argument yet for why builders and learners should stop anthropomorphizing AI.
- Andrej Karpathy Joins Anthropic: What It Tells Us About Where LLM Research Is Headed (opens in new tab)
The most-followed educator in AI just signed on to Anthropic's pre-training team. Here is what that move signals about the research directions that actually matter right now.
Key terms
Words you'll meet in this course
- Subword tokenization
- Splitting text into pieces smaller than words but larger than characters, so common words stay whole and rare words break into reusable parts.
- Embedding table
- A learned matrix with one row of numbers per vocabulary token; looking up a token's row gives the vector the model works with.
- Self-attention
- A step in which each token scores how relevant every other token in the sequence is, then mixes in their information according to those scores.
- Logits
- The raw scores a model produces for every token in its vocabulary before softmax turns them into probabilities.
- Softmax
- A function that turns a list of raw scores into probabilities that are all positive and add up to 1.
- Autoregressive generation
- Producing text one token at a time, where each new token is predicted from the prompt plus every token generated so far.
- Temperature
- A sampling setting that sharpens or flattens the next-token probabilities; lower values make output more predictable, higher values more varied.
- Hallucination
- Output that is fluent and confident but not supported by the input or by fact, such as an invented citation.
Help us choose
Which course should open next?
Pick the one you'd take first. Your vote helps us decide what to finish next.