Stage 1 · Lesson 3 of 6
Demonstrations, preferences and RLHF
are trained in stages. First a model learns from huge amounts of text. Then it is taught to follow instructions using : people write a prompt and the ideal answer, and the model is fine-tuned on them, called supervised (SFT).
Next, people compare two or more answers and pick the better one. Those choices form . A learns to predict which answer people prefer, and the chatbot is tuned against it.
That is reinforcement learning from human feedback (RLHF). OpenAI's InstructGPT paper in 2022 described exactly this, with hired labellers writing demonstrations and ranking outputs. Newer methods change the maths, but they still need human demonstrations and preferences.
Common trapthinking one careless comparison does not matter: thousands of them teach a model to prefer answers that only look good.
A question to check yourself, an AI practice session and 3 cheat sheet entries come with the full roadmap. Unlock
Resources for this stage
Start with· follow one of these
- CoursePaidGrokking Modern AI Fundamentals
Design Gurus · Beginner · $72 one-time
A structured course on how modern AI models are built and trained, for the full picture behind your tasks
- VideoFreeDeep Dive into Large Language Models like ChatGPT
Andrej Karpathy · Intermediate
Free: Karpathy walks through pre-training, fine-tuning and RLHF, including the role of human labellers