skills · How large language models work
How LLMs Actually Work
See what a language model computes, by building one.
Al terminar, podrás:
- A counting model that writes, and its loss on unseen text
- A byte-pair tokenizer trained from scratch
- Real token counts across languages
- Word vectors and nearest neighbors
- Masked multi-head attention in NumPy
- GPT-2's parameter count from its settings
- A training run that lowers loss
- A one-page note on how the model you use works
Desde $9/mes7,5 horas5 capítulos20 lecciones
Por ahora, los cursos están en inglés.
“Every reply is one guess at a time. Here is the guessing machine.”
Sobre este curso
Para quién es
- You use chat assistants and want to know what they compute, not only how to prompt them
- You can read a short Python script and would rather run the idea than read about it
- You want to check claims about models (size, context, why they get things wrong) for yourself
Lo que necesitas
- You can run a short Python script
- Python 3 on a laptop (NumPy and tiktoken are installed along the way)
Habilidades que practicarás
- Language models
- Tokenization
- Byte-pair encoding
- Embeddings
- Attention
- Transformers
- Python
- NumPy
Cuánto tiempo lleva
Unas 7,5 horas en total
- Capítulo 1 · Guessing the next token1,5 horas
- Capítulo 2 · Tokens1,5 horas
- Capítulo 3 · From ids to vectors1,5 horas
- Capítulo 4 · Attention1,5 horas
- Capítulo 5 · The whole machine1,5 horas
Qué incluye
- 20 lecciones
- 81 lecturas
- 20 ejemplos resueltos
- 40 ejercicios
- 40 comprobaciones rápidas
- 20 mitos comunes
- 20 cuestionarios
- 5 controles de fin de capítulo
Plan de estudios
5 capítulos, 20 lecciones
El curso completo: cada capítulo y lección. Toca un capítulo para expandirlo. Las lecciones se abren al empezar.
Carrera
Adónde puede llevarte este curso
Los roles con los que se relaciona este curso y las habilidades que podrás incluir en tu currículum y en LinkedIn cuando lo termines.
Habilidades que desarrollarás
Listas para pegar en tu currículum o en tu perfil de LinkedIn cuando termines el curso.
- Building and reading a bigram language modelExplain how a language model generates text one token at a time
- Decoding: greedy, sampling and temperatureExplain sampling temperature and its effect on model output
- Measuring data sparsity on held-out textEvaluate a model on held-out data
- Scoring a language model with loss and perplexityCompute cross-entropy loss and perplexity for a language model
- Comparing word, character and subword tokenizationExplain the trade-off between vocabulary size and sequence length
- Implementing byte-pair encodingImplement a BPE tokenizer in Python
- Counting tokens with tiktokenCount tokens for prompts and documents with a model's tokenizer
- Estimating token cost across languages and formatsEstimate token usage and cost for multilingual or structured inputs
- Building and sizing an embedding tableExplain how token ids map to embedding vectors
- Reading word-vector similarity with its limitsCompute cosine similarity and nearest neighbors in NumPy
- Explaining how position reaches a transformerExplain positional encodings in transformer models
- Computing logits, softmax and a baseline lossImplement softmax and cross-entropy loss in NumPy
- Computing attention as a weighted average of vectorsExplain how attention mixes token representations
- Implementing scaled dot-product attentionImplement query, key and value attention from the formula
- Applying and explaining the causal maskExplain autoregressive generation in decoder-style language models
- Sizing multi-head attention from a model configurationRead a transformer configuration for width and head count
- Reading a transformer block and counting a model's parametersEstimate model size from a published architecture configuration
- Training and scoring a small language modelExplain gradient descent and read a training log
- Reading chat templates and estimating prompt costExplain how a system prompt and chat history are sent to a language model
- Explaining language model behavior by mechanismExplain context windows, statelessness and hallucination to non-specialists
Preguntas
Antes de comprometerte
Empieza hoy